Exploiting Partial Assignments for Efficient Evaluation of Answer Set Programs with External Source Access

1 Introduction

HEX is an extension of Answer Set Programming (ASP) [Gelfond and Lifschitz, 1991] integrating external information sources (e.g. XML/RDF data bases, description logic reasoners, SAT solvers, etc.) using so-called external atoms. These pass information from the program, given by predicates and constants, to an external source, which returns output values to the program. For instance, the external atom &synonym[car ](X) might be used to find the synonyms X of car, e.g. automobile. HEX is highly expressive supporting e.g., nonmonotonic aggregates [Alviano et al., 2015] and recursive data exchange between programs and external sources.

Current evaluation algorithms for HEX [Eiter et al., 2012] first compute a complete truth-assignment by guessing the truth values of external atoms; only after the assignment is complete, calls to the external sources can check if the guesses were correct. This constitutes a major bottleneck and limits the applicability of HEX-programs because external sources are unable to guide the search algorithm effectively. Intuitively, evaluating external sources earlier, i.e., evaluating them under partial assignments, yields significant performance gains since it avoids many wrong guesses. On the other hand, such an evaluation has non-trivial issues because external sources are largely black boxes and may be nonmonotonic.

In this paper, we overcome the above problems by extending external sources from a Boolean semantics, defined only under complete assignments, to a three-valued semantics which is also defined under partial assignments. This enables novel evaluation techniques towards our main goal of efficiency, namely, (i) evaluating external sources already during the search with partial assignments, and based on this (ii) acquiring additional information about the external sources in the form of learned nogoods [Gebser et al., 2012] and (iii) minimizing the learned nogoods to approximate the semantics of external sources more closely. We note that the semantics of the overall formalism is not changed (answer sets are still two-valued). Moreover, our approach provides full backwards compatibility with existing two-valued sources.

⇤This research has been supported by the Austrian Science Fund (FWF) projects P27730 and W1255-N23.

Our techniques are related to theory propagation in SMT [Barrett et al., 2009] and minimization techniques in constraint ASP solvers such as CLINGCON [Ostrowski and Schaub, 2012]. These, however, usually rely on a tailored integration of theory solvers crafted by experts, whereas our approach allows a broad range of users, without prior knowledge on solver construction, to harness performance gained by the new learning techniques.

Furthermore, motivated by the formalism at hand, we develop refinements such as the simultaneous minimization of multiple nogoods, from which also related approaches might benefit. Although each additional step of evaluation and minimization is costly, we find in our experiments a good tradeoff and achieve a significant overall performance gain.

After necessary preliminaries in Section 2, we proceed to present our contributions as follows:

• ÷ In Section 3, we extend external atoms such that they can be evaluated under partial assignments. The output is then three-valued (true, false, unassigned), while answer sets are still two-valued. We also show how existing external atoms can be captured by our extension.
• ÷ Next, we present a novel evaluation algorithm which exploits the extension of external sources for earlier and significantly more effective search space pruning.
• ÷ In Section 4, we present nogood learning functions and a suitable minimization algorithm for HEX-programs, which exploit the generalized external atom interface for learning. We further show a close relation between nogood minimization and theory-specific learning.
• ÷ In Section 5, we present our prototypical implementation and evaluate our new techniques using a benchmark

suite. The results show a speedup of up to two orders of magnitude (an exponential gain is possible in theory).

• ÷ In Section 6, we discuss related work and conclude.

2 Preliminaries

We follow Drescher et al. [2008] for basic concepts. A (signed) literal is a positive or a negated ground atom Ta or Fa, where a is of form p(c1↪ ▷ ▷ ▷ ↪ c`) with predicate symbol p and constant symbols c1↪ ▷ ▷ ▷ ↪ c` from a finite set C, abbreviated as p(c); we write c 2 c if c = ci for some 1  i  `. For σ 2 ¶T↪ F♦ let σ = T if σ = F and σ = F if σ = T, and for a literal L = σa let L = σa. An assignment over the (finite) set A of atoms is a set A of literals s.t. for all a 2 A, Ta 2 A iff Fa ◁2 A; here Ta 2 A expresses that a is true and Fa 2 A that a is false.

A nogood is a set ¶L1↪ ▷ ▷ ▷ ↪ Ln♦ of literals Li↪ 1  i  n of type Ta or Fa. An assignment A is a solution to a nogood δ resp. a set of nogoods ∆, if δ 6✓ A resp. δ 6✓ A for all δ 2 ∆.

HEX-Programs. We briefly recall HEX-programs, which generalize (disjunctive) logic programs under the answer set semantics [Gelfond and Lifschitz, 1991]; for more details and background, see Eiter et al. [2005, 2014a].

Syntax. HEX-programs extend ordinary ASP programs by external atoms, which enable a bidirectional interaction between a program and external sources of computation. A ground external atom is of the form &g[p](c), where p = p1↪ ▷ ▷ ▷ ↪ pk is a list of input parameters (predicate names or object constants), called input list, and c = c1↪ ▷ ▷ ▷ ↪ cl are constant output terms. We restrict our theoretical investigation to ground programs as safety conditions allow for applying a grounding procedure.

Definition 1. A ground HEX-program ⇧ consists of rules

where each ai is a ground atom and each bj is either an ordinary ground atom or a ground external atom.

The body of r is B(r)=¶b1↪ ▷ ▷ ▷ ↪ bm↪ not bm+1↪ ▷ ▷ ▷ ↪ not bn♦.

Semantics. In the following, assignments are over the set A(⇧) of ordinary atoms that occur in the program ⇧ at hand. The semantics of a ground external atom &g[p](c) with k input and l output parameters wrt. an assignment A is given by the value of a 1+k+l-ary two-valued (Boolean) oracle function f&g that is defined for all possible values of A, p and c. Thus, &g[p](c) is true relative to A iff f&g(A↪ p↪ c) = T. Satisfaction of ordinary rules and ASP programs [Gelfond and Lifschitz, 1991] is then extended to HEX-rules and programs in the obvious way.

The answer sets of a HEX-program ⇧ are defined as follows. Let the FLP-reduct of ⇧ wrt. an assignment A be the set f⇧A = ¶r 2 ⇧ ♣ A ♣= b↪ for all b 2 B(r)♦ of all rules whose body is satisfied by A, and let for assignments A1, A2 denote A1  A2 that ¶Ta 2 A1♦ ✓ ¶Ta 2 A2♦. Then:

Definition 2. An assignment A is an answer set of a HEXprogram ⇧, if A is a -minimal model of f⇧A. 1

1 For ordinary ⇧, these are Gelfond & Lifschitz's answer sets.

Example 1. Consider the program ⇧ = ¶p &id[p]()♦, where &id[p]() is true iff p is true. Then ⇧ has the answer set A1 = ;; indeed it is a -minimal model of f⇧A1 = ;.

Evaluation. For evaluation, HEX-programs ⇧ can be transformed to ordinary ASP programs as follows. Each external atom &g[p](c) in ⇧ is replaced by an ordinary replacement atom e&g[p](c) and a rule e&g[p](c) _ne&g[p](c) is added. The answer sets of the resulting guessing program ˆ⇧ are computed by an ASP solver. But the assignment encoded by such an answer set may not satisfy ⇧, as f&g may evaluate &g[p](c) different from the guess for e&g[p](c). Thus, the answer set is merely a model candidate; if a check against the external sources finds no discrepancy, it is a compatible set. Formally:

Definition 3. A compatible set of a program ⇧ is an answer set ˆA of the guessing program ˆ⇧ such that f&g( ˆA↪ p↪ c) = T iff Te&g[p](c) 2 ˆA for all external atoms &g[p](c) in ⇧.

Each answer set of ⇧ is the projection of a compatible set to A(⇧), but not vice versa. To discard the non-answer sets, the evaluation algorithm calls an FLP check which uses unfounded sets to check minimality wrt. f⇧A [Eiter et al., 2014a].

Example 2 (cont'd). Reconsider ⇧ = ¶p &id[p]() ♦ from above. Then ˆ⇧ = ¶p e&id[p](); e&id[p] _ne&id[p] ♦ has the answer sets A1 = ; and A2 = ¶Tp↪ Te&id[p]♦. Here A1 is a -minimal model of f⇧A1 =;, but A2 not of f⇧A2 = ⇧.

3 Extension to Partial Assignments

In this section, we extend assignments and oracle functions to partial assignments, which provide for explicitly representing that some atom is yet unassigned.

Definition 4. A partial assignment over a set A of atoms is a set A of signed literals of kind Ta, Fa and Ua such that for all a 2 A exactly one of Ta 2 A, Fa 2 A or Ua 2 A holds.

Here, Ua denotes that the atom a is unassigned. A partial assignment not containing any Ua is a complete assignment.

For partial assignments A↪A0 we call A0 an extension of A, denoted A0 ⌫ A, if ¶Ta 2 A♦ [ ¶Fa 2 A♦ ✓ A0 (i.e., some unassigned atoms in A are flipped to true resp. false).

Next, we extend oracle functions in order to define the semantics of an external atom &g[p](c) wrt. partial assignments.

Definition 5. A three-valued oracle function f&g for a ground external atom &g[p](c) with k input and l output parameters is a 1+k+l-ary function such that f&g(A↪ p↪ c) 2 ¶T↪ F↪ U♦ for a partial assignment A and all possible values of p and c, and f&g(A↪ p↪ c) 6= U if A is complete.

Thus, &g[p](c) is true, false or unassigned relative to A, if the value of f&g(A↪ p↪ c) is T, F or U, respectively.

We require that once the output of f&g is assigned to true or false for some A, the value stays the same for all extensions.

Definition 6. A three-valued oracle function f&g is assignment-monotonic if f&g(A↪ p↪ c) = X, X 2 ¶T↪F♦, implies f&g(A0 ↪ p↪ c) = X for all assignments A0 ⌫ A.

Assignment-monotonicity guarantees that no compatible set is lost when querying external sources on partial assignments.

Example 3. Consider the program

It guesses, for a set of vertices v(X), all directed irreflexive subgraphs of the complete graph. Suppose &ge checks if the number of links is greater or equal to a natural number, s.t. the constraint restricts the number of arcs. The according two-valued oracle function f&ge(A↪ a↪ 2) for a complete assignment A evaluates to true if A contains at least two literals Ta(X↪ Y ), and to false otherwise. Extending the oracle to a partial assignment A0 is possible by defining an assignmentmonotonic three-valued oracle function f 0&ge (A0 ↪ a↪ 2) yielding true if ♣¶a(X↪ Y )♣Ta(X↪ Y ) 2 A0 ♦♣ ≥ 2, unassigned if ♣¶a(X↪ Y )♣Ta(X↪ Y ) 2 A0 or Ua(X↪ Y ) 2 A0 ♦♣ ≥ 2, and false otherwise.

Note that the definition of answer sets carries immediately over to programs with external atoms that use three-valued oracle functions. This is because answer sets are complete assignments and thus, the oracle function evaluates to T or F.

A two-valued oracle function, however, cannot handle partial assignments and is thus not a special case of a three-valued oracle function that can be passed to an algorithm expecting the latter. However, we can always obtain a three- from a twovalued oracle function such that answer sets remain invariant.

Proposition 1. For every HEX-program ⇧ and external predicate &g with a two-valued oracle function f&g(·↪ ·↪ ·) there is an external predicate &g 0 with assignment-monotonic threevalued oracle function f&g0(·↪ ·↪ ·) s.t. AS(⇧) = AS(⇧0 ), where ⇧0 results from ⇧ by replacing every &g by &g 0 .

Intuitively, we construct a three-valued oracle function which coincides with the two-valued one for complete assignments, and returns U otherwise. Hence, Proposition 1 allows us to "wrap" two-valued oracle functions for use by our algorithms below; in the implementation this is the basis for backwards compatibility with existing external sources.

We exploit partial assignments, by extending previous evaluation algorithms. In spirit of theory propagation in SMT solvers [Barrett et al., 2009], we use external theory learning (ETL). It is related to external behavior learning (EBL), which encodes observed output of external sources as nogoods [Eiter et al., 2012], but our extension works over partial assignments such that external sources may drive early propagation of truth values implied by the current partial assignment.

As for EBL, we can associate with each external source a learning-function ⇤ that yields a set of nogoods ⇤(&g[p]↪ A) learned from the evaluation of &g[p] under an assignment A. Learned nogoods have to be correct, i.e., they must not eliminate compatible sets. Formally, a nogood δ is correct wrt. a program ⇧, if all compatible sets of ⇧ are solutions to δ.

We extend learning functions for partial assignments as follows. Let E contain all expressions &g[p] that occur in ⇧ and S all signed (T↪ F↪U) literals on atoms in ˆ⇧.

Definition 7. A (three-valued) learning function for a HEXprogram ⇧ is a mapping ⇤ : E ⇥ 2S 7! 22S . It is called correct for ⇧, if all δ 2 ⇤(&g[p]↪A) are correct for ⇧, for all &g[p] in E and A 2 2S .

Algorithm 1: HEX-CDNL-PA

Input: A program ⇧ Output: An answer set of ⇧ if there exists one and ? otherwise Let ˆ⇧ be the guessing program of ⇧ ˆA ¶Ua ♣ a 2 A♦ // all atoms unassigned r ; // set of dynamic nogoods dl 0 // decision level while true do (a) ( ˆA↪r) Propagation(ˆ⇧↪r↪ ˆA) (b) if some nogood δ violated by ˆA then if dl = 0 then return ? Analyze conflict, add learned nogood to r, set dl to backjump level (c) else if ˆA is complete then A ˆA\ {Ta↪Fa ♣ a 2 A(ˆ⇧) if ˆA is not compatible or A is not a minimal model of f⇧A then r r[ ¶ˆA♦ else if there is an unfounded set U of ˆ⇧wrt. ˆA then Construct violated nogood for U and add it to r Analyze conflict, add learned nogood to r, set dl to backjump level else return A (d) else if Heuristics evaluates &g[y] and ⇤(&g[y]↪ ˆA) 6✓ r then r r[⇤(&g[y]↪ ˆA) (e) else Guess σa with σ 2 ¶T↪ F♦ for some variable a with Ua 2 ˆA dl dl + 1 ˆA ( ˆA∩ ¶Ua♦) [ ¶σa♦

Throughout the rest, we assume that learning functions are always correct for the programs at hand.

We now present a procedure for computing an answer set of a HEX-program shown in Algorithm 1. To compute multiple answer sets, we can naively add previous answer sets as constraints and call the algorithm again (cf. Gebser et al. [2007] for more elaborated techniques). The basic structure of Algorithm 1 resembles an ordinary ASP solver, but has additional checks in Part (c) and external calls to learn further nogoods in Part (d), which is based on partial assignments. Without the extensions, it computes an answer set ˆAof the guessing program ˆ⇧ and returns ˆA's projection to the atoms in ⇧ (cf. Drescher et al. [2008]). To this end, it starts from a void assignment and does unit propagation in Part (a) to derive further truth values. Part (b) backtracks and learns nogoods from conflicts; Part (c) without the first if (i.e., starting at the elsif-block) checks minimality wrt. the reduct of ˆ⇧. If no further truth values are set and the assignment is incomplete, it guesses in Part (e).

The if-block in Part (c) checks if ˆA is a compatible set (cf. Definition 3) and if ˆA's projection to A(⇧), i.e. A, is a minimal model of the FLP-reduct of ⇧ wrt. A. If both conditions are satisfied, A is an answer set of ⇧ (cf. Definition 2).

The additional calls to external sources and nogood learning in Part (d) are not mandatory but prune the search space; they may eliminate assignments violating known behavior of external sources already early in the search, while correctness of learning functions guarantees that no compatible set (hence no answer set) is eliminated. Notably and in contrast to previous algorithms [Eiter et al., 2012], external atoms are evaluated under partial assignments and use a three-valued oracle function. We can show that this algorithm is sound and complete:

Theorem1. If Algorithm 1 returns for ⇧ (i) an assignment A, then A is an answer set of ⇧ (ii) ?, then ⇧ is inconsistent.

4 Nogood Learning with Partial Assignments

In this section, we first discuss the generation of nogoods which encode parts of the semantics of external atoms. In contrast to previous work on external behavior learning (EBL), this works for partial assignments. When certain ground instances of an external atom can already be decided, nogoods can be learned early on and incompatible assignments can be identified; thus, they can guide the solver. Afterwards, we present a method for nogood minimization (relying on partial assignments and assignment-monotonicity), and show that this task and theory-specific learning are in fact closely related.

Three-valued Learning Functions. Let us first assume that we have no further knowledge about external sources and can only observe their (partial) output under a possibly partial input. We introduce a three-valued learning function for the general case, which is in fact a lifting of the according twovalued learning function defined by Eiter et al. [2012].

An input-output (io-)nogood is any nogood of formN = ¶σ1a1↪ ▷ ▷ ▷ ↪ σnan♦[¶σn+1e&g[p](c)♦ where σ1↪ ▷ ▷ ▷ ↪ σn+1 2 ¶T↪ F♦; we let NI = ¶σ1a1↪ ▷ ▷ ▷ ↪ σnan♦ be the literals over ordinary atoms (input part), NO = ¶σn+1e&g[p](c)♦ be the replacement atom (output part) of N, and σ(NO) = σn+1. We call N faithful, if f&e(A↪ p↪ c) = σ(NO) for all partial assignments A ◆ NI, i.e., it resembles the semantics of the external source. We note the following property:

Proposition 2. If N is a faithful io-nogood, then N is correct wrt. all programs.

As for the converse, correct nogoods (wrt. a given program) may be no faithful io-nogoods (or simply no io-nogoods).

When the oracle of an external atom is evaluated, the solver can create a new nogood for the observed input-output relationship. That is, evaluating &g[p] for a partial assignment A, the solver learns, given all true and false literals of input predicates, whether the output contains c, where f&g(A↪ p↪ c)6=U.

Definition 8. The learning function for an external predicate with input parameters &g[p] under partial assignment A is

where A0 = ¶σp(c) 2 A ♣ p 2 p↪ σ 6= U♦ is the relevant part of the external atom input.

Each respective nogood is an io-nogood by construction and as the oracle is assignment-monotonic, also faithful. Hence:

Proposition 3. Let &g[p](·) be an external atom in a HEXprogram ⇧. Then for all assignments A, the nogoods ⇤u(&g[p]↪ A) in Definition 8 are correct wrt. ⇧.

Example 4 (cont'd). Assume we add facts v(a) and v(b). Given the partial assignment A = ¶Tv(a)↪ Tv(b)↪ Fa(a↪ b)↪ Ua(b↪ a)↪ Tna(a↪ b)↪ Una(b↪ a)♦, the learning function ⇤u(&ge[a↪ 2]↪A) yields the single io-nogood ¶Fa(a↪ b)↪ Te&ge[a↪2]()♦, which is indeed faithful. For A = ¶Tv(a)↪ Tv(b)↪ Ta(a↪ b)↪ Ua(b↪ a)↪ Fna(a↪ b)↪ Una(b↪ a)♦, no ionogood can be learned and ⇤u(&ge[a↪ 2]↪A) returns the empty set since f 0&ge (A0 ↪ a↪ 2) = U.

We can refine a three-valued ⇤u similar to two-valued learning functions, cf. Eiter et al. [2012], and tailor it to external sources with specific properties. E.g., if an input predicate pi is monotonic, we can drop false atoms over pi from io-nogoods.

The general learning function ⇤u generates nogoods depending on the oracle function given a certain input. However, an external source provider usually knows the source semantics better and can thus provide better nogoods. The latter might include only the necessary atoms in the input; they are thus smaller and prune more of the search space. In such cases, it makes sense to provide custom learning functions ⇤l(&g[p]↪A) which generate for &g[p] and a (possibly partial) assignment A a set of nogoods.

Nogood Minimization. By exploiting three-valued semantics, we can also eliminate irrelevant (input) atoms from the nogoods in ⇤u, while faithfulness of io-nogoods is retained.

Definition 9. Given a faithful io-nogood N with NO = ¶σe&g[p](c)♦, the minimized nogoods of N are

This extends to sets S of nogoods by minimize(S) = SN2S minimize(N). Note that exponentially many minimal nogoods in the size of N are possible.

Example 5 (cont'd). For the assignment A = ¶Tv(a)↪ Tv(b)↪ Fa(a↪ b)↪ Fa(b↪ a)↪ Tna(a↪ b)↪ Tna(b↪ a)♦ and the faithful io-nogood N = ¶Fa(a↪ b)↪ Fa(b↪ a)↪ Te&ge[a↪2]()♦ 2 ⇤u(&ge[a↪ 2]↪ A), we obtain minimize(N) = ¶¶Fa(a↪ b)↪ Te&ge[a↪2]()♦↪ ¶Fa(b↪ a)↪ Te&ge[a↪2]()♦♦.

The minimized nogoods subsume all faithful io-nogoods.

Proposition 4. Let A be a partial assignment and N be a faithful io-nogood for &g[p] over the atoms in A. Then some N 0 2 minimize(⇤u(&g[p]↪A)) exists s.t. N 0 ✓ N.

As a subset of each faithful io-nogood occurs among all minimized nogoods, no further faithful io-nogoods prune the search space more effectively. Still there might be further correct nogoods (non-io ones and/or depending on the program).

In the following, we call a theory-specific learning function ⇤l(·↪ ·) complete for an external source &g, if for each partial assignments A0 ✓ A, input lists p, and output lists c the set ⇤l(&g[p]↪ A) is the least set such that f&g(A0 ↪ p↪ c) = σ with σ 2 ¶T↪F♦ implies A0 [ ¶σe&g[p](c)♦ is in ⇤l(&g[p]↪A); we call it partial otherwise. That is, ⇤l learns all and only io-nogoods with a premise over the current partial assignment which resemble the semantics of &g.

As it turns out, learning using complete theory-specific learning functions and nogood minimization are closely related. Let min✓(S) = ¶N 2 S ♣ @N 0 2 S s.t. N 0 ( N♦ be the restriction of S to subset-minimal nogoods. 2 Then:

Proposition 5. Let ⇤l be a complete theory-specific learning function for an external source &g. Then for all partial assignments A and input lists p it holds that minimize(⇤u(&g [p]↪ A)) = min✓(⇤l(&g [p]↪ A)).

2 Despite similar names, minimize differs from min✓ as it minimizes wrt. oracle results while min✓ just selects the minimal sets.

Input: A set S of faithful io-nogoods N with NI = ¶σ1a1↪ ▷ ▷ ▷ ↪ σnan♦ Output: A set of minimal faithful io-nogoods ch ; // cache for oracle calls for each signed literal σiai 2 NI do (a) for each io-nogood N 0 2 S with N 0O = ¶σn+1e&g[p](c)♦ do N s N 0I ∩ ¶σiai♦ // smaller oracle input (b) if hN s ↪ ·i 62 ch then ch ch [ {⌦N s ↪ ¶σe&g[p](c0 ) ♣ f&g(N s ↪ p↪ c0 )=σ6=U♦↵ (c) if σn+1e&g[p](c) 2 output for hN s ↪ outputi 2 ch then Replace N 0 by N s [ ¶σn+1e&g[p](c)♦ in S return S

This proposition implies that we have alternative techniques to learn all nogoods that prune the search space in an optimal (cf. Proposition 4) way. As above, it considers only faithful ionogoods while further correct nogoods may exist. Notably, the equality holds only under the premises of exhaustive minimization in the first case and a complete theory-specific learning function in the second; otherwise different sets of nogoods may be produced. As both operations are expensive and impractical, it makes sense to support both (incomplete) minimization and (incomplete) theory-specific learning functions.

In practice, we use Algorithm 2 to compute only one minimal io-nogood for each learned io-nogood. Instead of minimizing each nogood separately and to avoid redundant queries, we proceed in parallel and use a cache for the external atom output of a set S of io-nogoods with identical input but different outputs. The algorithm works by sequentially removing the same literal simultaneously from the premises of all N in S in Part (a), and checking whether the output for the resulting premises is already in the cache, in Part (b). If not, all outputs c0 s.t. f&g(A↪ p↪ c0 ) 6= U are computed (this is a single call in the implementation) and stored in the cache. Otherwise, no external source call is needed. It is then checked if the resulting nogood is still faithful in Part (c), and N is replaced by its reduced equivalent in S in this case. Formally:

Proposition 6. For a set S of faithful io-nogoods with equal input parts and distinct output parts, Algorithm 2 yields exactly one faithful io-nogood N 0 2 minimize(N) for each N 2 S.

5 Implementation and Evaluation

For experimentation, we integrated our techniques into the reasoner DLVHEX with GRINGO 4.4.0 and CLASP 3.1.1 as backends. All benchmarks were run on a Linux machine with two 12-core AMD Opteron 6176 SE CPUs and 128 GB RAM; timeout was 300 secs and memout 8 GB per instance. We used the HTCondor load distribution system (http://research.cs.wisc. edu/htcondor) to ensure robust runtimes (i.e., deviations of runs on the same instance are negligible). Average runtimes of 50 instances per problem size for computing all answer sets are reported in seconds, 3 with timeouts in parentheses, and the average number of solutions, where '≥' respects timeouts.

Benchmark Configurations. We used learning function ⇤u and compared 5 configurations. We first tested 3 heuristics for additional external source calls (cf. Algorithm 1, Part (d)) without nogood minimization, viz. never (no additional calls, i.e. DLVHEX without the new techniques), periodic (partial evaluation at each 10 th heuristics call), and always (partial evaluation at every heuristics call). We then tested nogood minimization instead of additional calls (i.e., only for complete assignments): minimizing all nogoods (ngm) resp. the currently conflicting ones (ngmc). We omit results for minimization combined with periodic or always, which was always significantly slower than some other configuration (due to many more external calls with little gain).

3 Results for computing the first answer sets are comparable.

Our hypothesis was that periodic and always decrease the runtime over never if useful information is obtainable by early evaluation, and increase it otherwise. We expected the tradeoff to be mitigated by just minimizing nogoods on complete assignments using ngm instead of evaluating early. We further expected that restricting the minimization to nogoods that directly trigger backjumping, as tested in ngmc, is even more effective due to reduced overhead. It was expected that partial evaluation performs better for instances with a small percentage of solutions wrt. the assignments in the search space since a large percentage leaves less room for pruning it.

Pseudo-Boolean Problems. Pseudo-boolean problems constitute sets of pseudo-boolean constraints (PBCs) of the form C0p0 + ▷▷▷ + Cn-1pn-1 ≥ Cn, where all pi are literals and all Ci are integers [E´en and S¨orensson, 2006]. Our implementation guesses an interpretation of the pi by disjunctive rules and has a constraint not&pbCheck[p↪ pbInst]() which is satisfied wrt. a complete assignment to p iff the latter represents a solution for the PBCs. The strict separation of the guess and the check part results in benchmark instances that are well-suited for investigating the effect of a tighter integration of the solving algorithm and the evaluation of external constraints. 4 We extend the semantics of the external atom to

4 Note that for the purpose of solving PBCs as part of a HEXprogram (possibly in combination with other external sources), the external source could directly interface a dedicated PB solver.

20 300.00 (50) 142.66 (13) 174.44 (18) 300.00 (50) 180.63 (16) 300.00 (50) ≥3918.82

drives(X↪ Y ) driver(X)↪ customer(Y )↪DL[; isIn](X↪A)↪ DL[; isIn](Y↪ A)↪ region(A)↪ not ndrives(X↪ Y )▷ ndrives(X↪ Y ) driver(X)↪ customer(Y )↪ not drives(X↪ Y )▷ drives(X↪ Y )↪ drives(X1↪ Y )↪ X 6= X1▷ drivesECust(X↪ Y ) drives(X↪ Y )↪ DL[eDrives ] drivesECust ; ECust ](Y )▷ driven(Y ) drives( ↪ Y )▷ not driven(Y )↪ customer(Y )▷ #count¶Y : drives(X↪ Y )♦ > 4↪ driver(X)▷ drives(X↪ Y )↪ not DL[eDrives ] drivesECust ; ECust ](Y )↪ DL[eDrives ] drivesECust; EDriver](X)▷ drives(X↪ Y )↪ DL[eDrives ] drivesECust ; ECust ](Y )↪ not DL[eDrives ] drivesECust ; EDriver ](X)▷

Figure 1: Taxi Assignment Rules

partial assignments A as follows: &pbCheck [p↪ pbInst]() is true wrt. A if every input PBC fulfills PTpi2A Cipi ≥ Cn; it is false, if some input PBC fulfills PFpi62A Cipi < Cn; it is unassigned otherwise.

First, we tested randomly generated problems with N 2 [4↪ 36] variables and 4 ⇥N PBCs with n = 6 and Ci 2 [1↪ 5] for 0  i  n (Table 1). The ratio between N and the number of constraints ensures that only a small fraction of all assignments are answer sets. A clear improvement over never is observed whenever partial evaluation is used. Without nogood minimization, always shows the best performance, with periodic falling in-between always and never, hence learning the io-behavior of the external source as early as possible outweighs the runtime overhead for querying it additionally. Overall, ngmc shows the best performance.

Second, to investigate the behavior when large parts of the search space contain solutions, we fixed the number of variables and PBCs to 15 and 60, resp., and tested different lengths n 2 [4↪ 20] (Table 2). The solution count increases with length, and for n > 14 nearly all assignments are answer sets. Expectedly, periodic and always are slower than never if many (more than about half of) the candidates are solutions. Frequent evaluation is detrimental here, as runtime investment has no pay-off in information gain or early search termination. Likewise, minimizing all io-nogoods is worse as identical nogoods are computed for many complete assignments; ngmc, however, is very efficient with only a small runtime overhead, because it focuses on valuable (i.e. conflicting) io-nogoods.

Taxi Assignment. We use a HEX-program to assign taxi drivers to customers under constraints. Naturally, a customer and her driver must be in the same region; customers may share the driver, and a taxi fits at most 4 customers. A descrip-

50 300.00 (50) 300.00 (50) 261.09 (32) 170.16 (17) 158.21 (12) ≥2850.36

Table 4: Conflicting Strategic Companies Results

tion logic (DL) ontology has information such as locations of individuals, e-customers (customers demanding electric cars), and e-drivers (drivers of electric cars); e-customers must be assigned to e-drivers, and normal customers to normal drivers.

The answer sets of the program with facts driver(d), customer(c) and region(r) for drivers d, customers c and regions r, and the rules in Figure 1 encode legal assignments. The ontology is queried using DL-atoms [Eiter et al., 2008], which offer a more tailored syntax for specific external atoms; e.g. DL[eDrives ] drivesECust;EDriver](X) retrieves all individuals in the class EDriver, where "eDrives ] drivesECust" enhances the role eDrives with the true atoms of drivesECust before evaluation. DL-atoms are implemented in the DL-Lite plug-in for DLVHEX [Eiter et al., 2014b]. Exploiting monotonicity of DLs, the threevalued evaluation under a partial assignment A is as follows: it returns true, if the query is true with the minimal addition of assertions (in the example ¶eDriver(d↪ d0 ) ♣ TdrivesECust(d↪ d0 ) 2 A♦); false, if the query is not true with the maximal possible addition (¶eDriver(d↪ d0 ) ♣ FdrivesECust (d↪ d0 ) 62 A♦); and unassigned otherwise.

In our tests, we increased the number N of drivers and customers from 4 to 20, which were put in N◁2 regions randomly, with the drivers balanced among regions; half of the customers were e-customers (Table 3). As DL-atoms have output constants, we could also compare simultaneous and sequential nogood minimization (ngm-sq). The results for partial evaluation using periodic, always and ngmc are mixed and there is no clear winner; however, all of them are significantly faster than never. Indeed, the external DL calls are costly, and waiting a bit until the next one can pay off. As the premise of an io-nogood can be large but the output often depends only on a small part, minimization can drastically shrink io-nogoods. However, the cost is many external calls, which can be reduced by minimizing nogoods with same premise simultaneously; in total, a significant speedup results.

Conflicting Strategic Companies. Strategic Companies is a popular problem in ASP. We use the encoding schema of Leone et al. [2006], with an additional constraint that checks for conflicts on strategic sets via the external atom &conflict[s](), e.g. imposing legislative restrictions. On complete assignments, it is true if companies ci↪ cj in s are related by an external conflict relation R ✓ C⇥C, and false otherwise. We use a three-valued oracle function that returns true for a partial assignment A if Ts(ci)↪ Ts(cj) 2 A holds for some (ci↪ cj) 2 R, false if Fs(ci) 2 A or Fs(cj) 2 A for every (ci↪ cj) 2 R, and unassigned otherwise.

Since finding a strategic set is computationally hard, excluding candidate strategic sets with a conflict early in the search by partial evaluations should noticeably decrease the runtime.

We ran tests on instances with N 2 [10↪ 50] companies, at most N randomly assigned control relations, 5 ⇥N products with randomly assigned producers, and N◁2 randomly created conflicts (Table 4); the latter cut more than 90% of the strategic sets (i.e., solution candidates). Partial evaluation always decreases the runtime, where ngmc shows the best results. Notably, for computing strategic sets containing a specific company (which is ⌃P2 -hard) we obtain similar results.

Instances and details on the experiments can be found at http: //www.kr.tuwien.ac.at/research/projects/inthex/partialevaluation.

6 Discussion and Conclusion

RelatedWork. Our work is most closely related to constraint ASP solving [Gebser et al., 2009] and SMT (in particular theory propagation) [Nieuwenhuis et al., 2006]. However, our approach is more general as it supports arbitrary external sources, even if they may be semantically black boxes; thus, the context for learning also differs. While theory literals and constraint atoms are directly related via a theory and shared constraint variables, respectively, external atoms are only indirectly related via their input. Hence, ETL is realized by learning io-relations in our case, while values of theory literals and constraint atoms can be propagated within the solver. Also Ostrowski and Schaub [2012] considered nogood minimization, using different algorithms that avoid expensive resets of the constraint solver. In our more general setting, this is inapplicable, while other issues arise; e.g., that external atoms can have multiple output values motivates our algorithm for simultaneous io-nogood minimization. Overall, the mentioned approaches do not lend themselves to a direct comparison.

Antic et al. [2013] considered partial HEX-semantics before, employing Approximation Fixpoint Theory (AFT). Although, we only consider two-valued answer sets and do not apply a fixpoint construction, our partial oracle functions coincide with their three-valued ones. Similarly, Pelov et al. [2004] have defined a family of partial stable model semantics for logic programs with aggregates using AFT. Assignment-monotonic oracle functions are also related to their approximating aggregate relations which must be precision-monotone and generalize ordinary aggregate relations to a three-valued semantics.

Typical integration schemas for SMT have been identified [Balduccini and Lierler, 2013b], which are also applicable to ASP modulo theories; a comparison is given in [Balduccini and Lierler, 2013a]. In black-box integration, the SAT solver blindly generates a model and passes it for checking to the theory solver. If it passes the check, it is returned, otherwise it is added as a constraint to the instance and the solver restarts. This allows for easy coupling with arbitrary theories but does not enable search space pruning. In grey-box integration, the theory solver is only called for complete models of the SAT instance, but the SAT solver is merely suspended during checking and can continue its search afterwards; integration is still relatively simple. Only in clear-box integration, the SAT solver is interleaved with the theory solver, which is called already for partial assignments and in turn may propagate further truth values or detect inconsistencies. However, the integration is much more challenging as the theory solver must identify atoms implied by the given partial assignment, or inconsistency reasons, respectively. Applied to HEX, the grey-box schema corresponds to the algorithms before EBL [Eiter et al., 2012] was introduced (black-box integration, i.e., complete restarts, was never used). With EBL, the algorithms followed an intermediate schema between grey- and clear-box: external sources were still only evaluated under complete assignments but the learned nogoods possibly pruned the search space.

Conclusion. The techniques introduced in this paper yield full-fledged clear-box integration. Moreover, due to automatic nogood minimization, developers of external sources do not need to manually describe implied truth values or inconsistency reasons, but only need to implement a three-valued oracle function, which keeps the integration of sources simple.

In our experiments, the new techniques yield a speedup of up to two orders of magnitude; unsurprisingly, their ranking depends on the instances. This is similar to observations by Ostrowski and Schaub [2012], who reported mixed results for different propagation delays. Our results are also in line with results in SMT, where theory propagation, if doable with small overhead, is crucial for performance [Dutertre and de Moura, 2006; Lahiri et al., 2006; Nieuwenhuis and Oliveras, 2005]. We observed that in most cases learning from complete assignments plus minimization of conflicting nogoods (based on partial assignments) outperforms learning during search; hence, this setting is suggestive as a default. This is explained by the fact that in this case, learning focuses on nogoods that are useful for conflict resolution, thus the information gain is similar and the overhead much smaller. This matches the observation by Nieuwenhuis et al. [2006] that conflict analysis uses only a small fraction of the lemmas learned by theory propagation, which can be addressed with lazy explanations [Gent et al., 2010]. The speedup can be up to exponential, as evidenced by an external atom whose truth value is definite after assigning a single input atom, e.g. &empty[p]() to check if an atom over p is true. Each naive nogood eliminates one of exponentially many assignments, but a linear number of minimized ones eliminates all wrong guesses.

Although our minimization algorithm cannot be improved in the worst case (the nogood is already minimal), more sophisticated algorithms for computing minimal conflicts have been considered in the literature, which are likely to improve the performance when nogoods comprise many irrelevant literals. E.g., the QUICKXPLAIN algorithm [Junker, 2004] employs a divide-and-conquer strategy to find relaxations of overconstrained problems, which allows pruning parts of a binary search tree not containing relevant constraints. Based on this, Shchekotykhin et al. [2015] developed the MERGEXPLAIN algorithm, capable of efficiently determining multiple minimal conflicts during one problem analysis run, which could be integrated into our approach for obtaining all minimal io-nogoods.

Topics for ongoing and future work include further heuristics for external evaluation (e.g., evaluating or skipping based on the past information gain), further nogood-minimization strategies using knowledge about external sources (e.g., functionality or convexity), and exploiting partial assignments also in the minimality check of answer set candidates.