NeuPSL: Neural Probabilistic Soft Logic

1 Introduction

The field of artificial intelligence (AI) has long sought a symbiotic union of neural and symbolic methods. Neural-based methods excel at low-level perception and learn from large training data sets but struggle with interpretability and generalizing in low-data settings. Meanwhile, symbolic methods can effectively use domain knowledge, context, and common sense to reason with limited data but have difficulty representing complex low-level patterns. Recently, neuro-symbolic computing (NeSy) [Besold et al., 2017; d'Avila Garcez et al., 2019; De Raedt et al., 2020] has emerged as a promising new research area with the goal of developing systems that integrate neural and symbolic methods in a mutually beneficial manner.

A neural and symbolic union has the potential to yield two highly desirable capabilities - the ability to perform structured prediction (joint inference) across related examples that possess complex low-level features and the ability to jointly learn (joint learning) and adapt parameters over neural and symbolic models simultaneously. For instance, predicting the result of competitions between teams using historical performance statistics in a tournament bracket requires methods to perform joint inference to reason over low-level trends and avoid inconsistencies such as two first-place finishes. Unfortunately, joint inference problems quickly grow in complexity as the output space typically increases combinatorially. For example, in the tournament setting, as the number of entries increases, the number of potential solutions grows exponentially (O(2n )). An open challenge in the NeSy community is scaling joint inference and reasoning.

This paper introduces Neural Probabilistic Soft Logic (NeuPSL), a novel NeSy method that integrates deep neural networks with a symbolic method designed for fast joint learning and inference. NeuPSL extends probabilistic soft logic (PSL) [Bach et al., 2017], a state-of-the-art and scalable probabilistic programming framework that can reason statistically (using probabilistic inference) and logically (using soft rules). PSL has been shown to excel in a wide variety of tasks, including natural language processing [Beltagy et al., 2014; Deng and Wiebe, 2015; Liu et al., 2016; Rospocher, 2018], data mining [Alshukaili et al., 2016; Kimmig et al., 2019], recommender systems [Kouki et al., 2015], knowledge graph discovery [Pujara et al., 2013], fairness modeling [Farnadi et al., 2019; Dickens et al., 2020], and causal reasoning [Sridhar et al., 2018]. The key innovation of NeuPSL is a new class of predicates that rely on neural network output for their values. This change fundamentally alters the learning and joint inference problems by requiring efficient integrated symbolic and neural parameter learning. The appeal of this extension is that it allows for the semantics and implementation of the symbolic language to remain the same as PSL, while also incorporating the added benefit of low-level neural perception. To gain a deeper understanding of optimizing the symbolic and neural parameters, we propose a versatile mathematical framework, Neuro-Symbolic Energy-Based Models (NeSy-EBMs), that enables many NeSy systems to utilize established Energy-Based Model learning losses and algorithms. Utilizing this theory and leveraging the unique relaxation properties of PSL, we show that a gradient over these neural predicates can be calculated and passed back to common back-propagation engines such as PyTorch or Tensorflow, allowing for scalable end-to-end gradient training.

Our key contributions include: 1) We define NeuroSymbolic Energy-Based Models (NeSy-EBMs), a family of energy-based models, and show how they provide a foundation for describing, understanding and comparing NeSy systems. 2) We introduce NeuPSL, describe how it fits into the NeSy ecosystem and supports scalable joint inference, and show how it can be trained end-to-end using a joint energybased learning loss. 3) We perform extensive evaluations over two image classification tasks and two citation network datasets. Our results show NeuPSL consistently outperforms existing approaches on joint inference tasks and can more efficiently leverage structure, particularly in low-data settings.

2 RelatedWork

Neuro-symbolic computing (NeSy) is an active area of research that aims to incorporate logic-based reasoning with neural networks [d'Avila Garcez et al., 2002; Bader and Hitzler, 2005; d'Avila Garcez et al., 2009; Serafini and d'Avila Garcez, 2016; Besold et al., 2017; Donadello et al., 2017; Yang et al., 2017; Evans and Grefenstette, 2018; Manhaeve et al., 2021; d'Avila Garcez et al., 2019; De Raedt et al., 2020; Lamb et al., 2020; Badreddine et al., 2022]. The advantages of NeSy systems include interpretability, robustness, and the ability to integrate various sub-problem solutions (such as perception, reasoning, and decision-making). For a thorough introduction to NeSy literature, we refer the reader to the excellent surveys by Besold et al. (2017) and De Raedt et al. (2020). In this section, we identify key NeSy research categories and provide a brief description of each.

Differentiable frameworks of logical reasoning: Methods in this category use neural networks' universal function approximation properties to emulate logical reasoning inside networks. Examples include: Rockt¨aschel and Riedel (2017), Boˇsnjak et al. (2017), Evans and Grefenstette (2018), and Cohen et al. (2020).

Constrained Output: These approaches enforce constraints or regularizations on the output of neural networks. Examples include: Hu et al. (2016), Diligenti et al. (2017), Donadello et al. (2017), Mehta et al. (2018), Xu et al. (2018), and Nandwani et al. (2019).

Executable logic programs: These approaches use neural models to build executable logical programs. Examples include Liang et al. (2017) and Mao et al. (2019). We highlight Logic Tensor Networks (LTNs) [Badreddine et al., 2022], as we include this approach in our empirical evaluation. LTNs connect neural predictions into functions representing symbolic relations with real-valued or fuzzy logic semantics.

Neural networks as predicates: This line of work integrates neural networks and probabilistic reasoning by introducing neural networks as predicates in the logical formulae. This technique provides a very general and flexible framework for NeSy reasoning and allows for the use of multiple networks as well as the full incorporation of constraints and relational information. Examples include DASL [Sikka et al., 2020], NeurASP [Yang et al., 2020], Nuts&Bolts [Sachan et al., 2018], DeepProbLog (DPL) [Manhaeve et al., 2021], and our proposed method (Neural Probabilistic Soft Logic). DPL

combines general-purpose neural networks with the probabilistic modeling of ProbLog [De Raedt et al., 2007] in a way that allows for learning and inference over complex tasks, such as program induction. We include DPL in our empirical evaluation.

3 Neuro-Symbolic Energy-BasedModels

With the success and growth of NeSy research, there is an increasing need for a common formalization of NeSy systems to accelerate the research and understanding of the field. We fill this need with a general mathematical framework, NeuroSymbolic Energy-Based Models (NeSy-EBMs). NeSy-EBMs encompass previous approaches and establishes the foundation of our approach. Energy-Based Models (EBMs) [LeCun et al., 2006] measure the compatibility of a collection of observed (or input) variables x ∈ X and target (or output) variables y ∈ Y with a scalar-valued energy function: E : Y ×X → R. Low energy states of the variables represent high compatibility. Prediction or inference in EBMs is performed by finding the lowest energy state of the variables y given x. Energy functions are parameterized by variables w ∈ W, and learning is the task of finding a parameter setting that associates low energy to correct solutions.

Building on the well-known EBM framework, NeSyEBMs are a family of EBMs that integrate neural architectures with explicit encodings of symbolic relations. The input variables are organized into neural, xnn ∈ Xnn, and symbolic, xsy ∈ Xsy, vectors. Furthermore, the parameters of the energy function, w, are partitioned into neural weights, wnn ∈ Wnn, and symbolic weights, wsy ∈ Wsy. Formally,

Definition 1 (NeSy-EBM). Let y ∈ Y and xsy ∈ Xsy be vectors of variables with symbolic interpretations. Let gnn be neural networks with neural weights wnn ∈ Wnn and inputs xnn ∈ Xnn. A symbolic potential is a function of y, xsy, and gnn(·) parameterized by symbolic weights wsy ∈ Wsy: ψ(y, xsy,wsy, gnn(xnn,wnn)) ∈ R. A NeSy-EBM energy function is a mapping of a vector of m symbolic potential outputs, Ψ(y, xsy,wsy, xnn,wnn) = [ψi(y, xsy,wsy, gnn(xnn,wnn))]mi=1 , to a real value: E(Ψ(y, xsy , wsy , xnn, wnn)) ∈ R.

NeSy-EBMs are differentiated from one another by the instantiation process, the form of the symbolic potentials, and the definition of the energy function. In appendix, we formally show how two NeSy systems DeepProbLog (DPL) [Manhaeve et al., 2018] and Logic Tensor Networks (LTNs) [Badreddine et al., 2022] fit into the NeSy-EBM framework. In summary, DPL uses neural network outputs to specify event probabilities that are used in logical formulae defining probabilistic dependencies. The definition of the DPL symbolic potentials and energy function are tied to the inference task; a different definition of the symbolic potential and energy function is used to implement marginal versus MAP inference. For marginal, the most common DPL inference, symbolic potentials are functions of marginal probabilities, and the energy function is a joint distribution that is the sum of the symbolic potentials. LTNs instantiate a model which forwards neural network predictions into functions representing symbolic relations with real-valued or fuzzy logic semantics. The fuzzy logic functions are symbolic potentials that are aggregated to define the energy function. The following section will introduce how our approach, NeuPSL, is instantiated as a NeSy-EBM. Using this common framework, understanding and theoretical advances can be made across NeSy approaches.

3.1 Joint Reasoning in NeSy-EBMs

We highlight two important categories of NeSy-EBM energy functions: joint and independent. Formally, an energy function that is additively separable over the output variables y is an independent energy function, i.e., corresponding to each of the ny components of the output variable y there exists functions ny functions E1(y[1], xsy, wsy, g(xnn, wnn)), · · · , Eny (y[ny], xsy, wsy, g(xnn, wnn)) such that

While a function that is not separable over output variables y is a joint energy function. This categorization allows for an important distinction during inference and learning. Independent energy functions simplify inference and learning as finding an energy minimizer, y∗ , can be distributed across the independent functions Ei. In other words, the predicted value for a variable y[i] has no influence over that of y[j] where j ̸= i and can therefore be predicted separately, i.e., independently. However, independent energy functions cannot leverage some joint information that may be used to improve predictions. See appendix for further details.

4 Neural Probabilistic Soft Logic

Having laid the NeSy-EBM groundwork, we now introduce Neural Probabilistic Soft Logic (NeuPSL), a novel NeSyEBM framework that extends the probabilistic soft logic (PSL) framework [Bach et al., 2017]. At its core, NeuPSL leverages the power of neural networks' low-level perception by seamlessly integrating their outputs with a collection of symbolic potentials generated through a PSL program. Figure 1 provides a graphical representation of this process. The symbolic potentials and neural networks together define a deep hinge-loss Markov random field (Deep-HL-MRF), a tractable probabilistic graphical model that supports scalable convex joint inference. This section provides a comprehensive description of how NeuPSL instantiates its symbolic potentials and how the symbolic potentials are combined to define an energy function, while the following section details NeuPSL's end-to-end neural-symbolic inference, learning, and joint reasoning processes.

NeuPSL instantiates the symbolic potentials of its energy function using the PSL language where dependencies between relations and attributes of entities in a domain, defined as atoms, are encoded with weighted first-order logical clauses and linear arithmetic inequalities referred to as rules. To illustrate, consider a setting in which a neural network is used to classify the species of an animal in an image. Further, suppose there exists external information suggesting when two images may contain the same entity. The information linking the images may come from various sources, such as the images' caption or metadata indicating the images were captured by the same device within a short period of time. NeuPSL represents the neural network's animal classification of an image (Image1) as a species (Species) with the atom NEURAL(Image1, Species) and the probability that two images (Image1 and Image2) contain the same entity with the atom SAMEENTITY(Image1, Image2). Additionally, we represent NeuPSL's classification of Image2 with CLASS(Image2, Species). The following weighted logical rule in NeuPSL represents the notion that two images identified as the same entity may also be of the same species:

The parameter w is the weight of the rule, and it quantifies its relative importance in the model. Note these rules can either be hard or soft constraints. Atoms and weighted rules are templates for creating symbolic potentials or soft constraints. To create these symbolic potentials, atoms and rules are instantiated with observed data and neural predictions. Atoms instantiated with elements from the data are referred to as ground atoms. Then, valid combinations of ground atoms substituted in the rules create ground rules. To illustrate, suppose that there are two images {Id1, Id2} and three species classes {Cat,Dog, Frog}. Using the above data for cats would result in the following ground rules (analogous ground rules would be created for dogs and frogs):

Ground atoms are mapped to either an observed variable, xsy,i, target variable, yi, or a neural function with inputs xnn and parameters wnn,i: gnn,i(xnn,wnn,i). Then, variables are aggregated into the vectors xsy = [xsyi ]nx i=1 and y = [yi]ny i=1 and neural outputs are aggregated into the vector gnn = [gnn,i]ng i=1 . Ground rules are either logical (e.g., Equation 1) or arithmetic defined over xsy, y, and gnn. These ground rules create one or more potentials ϕ(·) ∈ R, where logical rules are relaxed using Łukasiewicz continuous valued logical semantics [Klir and Yuan, 1995]. Each potential ϕ(·) is associated with a weight wpsl inherited from its instantiating rule. The potentials and weights from the instantiation process are used to define a member of a tractable class of graphical models, deep hinge-loss Markov random fields (Deep-HL-MRF):

Definition 2 (Deep Hinge-Loss Markov Random Field). Let y ∈ [0, 1]ny and xsy ∈ [0, 1]nx be vectors of [0, 1] valued variables. Let gnn = [gnn,i]ng i=1 be functions with corresponding parameters wnn = [wnn,i]ng i=1 and inputs xnn. A deep hinge-loss potential is a function of the form

where l(·) is a linear function and α ∈ {1, 2}. Let T = [ti]ri=1 denote an ordered partition of a set of m deep hingeloss potentials: {ϕ1, · · · , ϕm}. For each partition ti define Φi(y, xsy, xnn,wnn) := ∑j∈ti ϕi(y, xsy, xnn,wnn) and let Φ(y, xsy, xnn,wnn) := [Φi(y, xsy, xnn,wnn)]ri=1 . Further, let wpsl = [wpsl,i]ri=1 be a vector of non-negative weights corresponding to the partition T . Then, a deep hinge-loss energy function is

Further, let c = [ci]qi=1 be a vector of q linear constraints in standard form, defining the feasible set Ω = {y, xsy | ci(y, xsy) ≤ 0, ∀i ∈ {0, · · · , q}}. Then a deep hinge-loss Markov random field, P, with random variables y conditioned on xsy and xnn is a probability density of the form

Deep-HL-MRFs naturally fit into the NeSy-EBM framework. The symbolic potentials of deep-HL-MRFs are the aggregated and scaled deep hinge-loss potentials:

Then the energy function is the sum of symbolic potentials:

ENeuP SL(y, xsy , xnn, wnn, wpsl)

5 NeuPSL Inference and Learning

There is a clear connection between neural and symbolic inference in NeuPSL that allows any neural architecture to interact with symbolic reasoning in a simple and expressive manner. The NeuPSL neural-symbolic interface and inference pipeline is shown in Figure 1. Neural inference is computing the output of the neural networks given the input xnn, i.e., computing gnn,i(xnn, wnn,i) for all i. NeuPSL symbolic inference minimizes the energy function over y:

Note that the hinge-loss potentials are convex in y and hence, with the common constraint enforcing symbolic parameters to be non-negative, i.e., wpsl > 0, the energy function is convex in y. Any scalable convex optimizer can be applied to solve (6). NeuPSL uses the alternating direction method of multipliers [Boyd et al., 2010].

NeuPSL learning is the task of finding both neural and symbolic parameters, i.e., rule weights, that assign low energy to correct values of the output variables and higher energies to incorrect values. Learning objectives are functionals mapping an energy function and a set of training examples S = {(yi, xsy,i, xnn,i) : i = 1, · · · , P} to a real-valued loss. As the energy function for NeuPSL is parameterized by the neural weights wnn and symbolic weights wpsl, we express the learning objective as a function of wnn, wpsl, and S: L(S, wnn, wpsl). Learning objectives follow the standard empirical risk minimization framework and are therefore separable over the training examples in S as a sum of per-sample loss functions Li(yi, xi, xnn,i,wnn,wpsl). Concisely, NeuPSL learning is the following minimization:

In the learning setting, variables yi from the training set S are partitioned into vectors yi,t and zi. The variables yi,t represent variables for which there is a corresponding truth value, while zi represent latent variables. Without loss of generality, we write yi = (yi,t, zi).

There are multiple losses that one could motivate for optimizing the parameters of an EBM. Common losses, including the loss we present in this work, use the following terms:

In words, z∗i and y∗ i are the lowest energy states given (yi,t, xsy,i, xnn,i) and (xsy,i, xnn,i), respectively. A special case of learning is when the per-sample losses are not functions of z∗i and y∗ i , and more specifically, the losses do not require any subproblem optimization. We refer to this situation as constraint learning. Constraint learning reduces the time required per iteration at the cost of expressivity.

All interesting learning losses for NeuPSL are a composition of the energy function. Thus, a gradient-based learning algorithmwill require the following partial derivatives: 1

Continuing with the derivative chain rule and noting the potential can be squared (α = 2) or linear (α = 1), the potential partial derivative with respect to wnn[i] is the piece-wise defined function: 1

Since l(y, xsy, gnn(xnn,wnn)) is a linear function, the partial gradient with respect to gnn[i] is trivial. With the partial derivatives presented here, standard backpropagation-based algorithms for computing gradients can be applied for both neural and symbolic parameter learning.

Energy Loss: A variety of differentiable loss functions can be chosen for L. For simplicity, in this work, we present the energy loss. The energy loss parameter learning scheme directly minimizes the energy of the training samples, i.e., the per-sample losses are:

Notice that inference over the latent variables is necessary for gradient and objective value computations. However, a complete prediction fromNeuPSL, i.e., inference over all components of y, is unnecessary. Therefore the parameter learning problem is as follows:

With L2 regularization, the NeuPSL energy function is strongly convex in all components of yi. Thus, by Danskin (1966), the gradient of the energy loss, Li(·), with respect to wpsl at yi, xi, xnn,iwnn is:

Then the per-sample energy loss partial derivative with respect to wnn[j] at yi, xsy,i, xnn,i,wpsl is:

1 Note arguments of the energy function and symbolic potentials are dropped for simplicity, i.e., E(·) = E(y, xsy,i, xnn,i,wnn,wpsl), ϕ(·) = ϕ(y, xsy, xnn,wnn), and gnn[i](·) = gnn[i](xnn, wnn).

Details on the learning algorithms and accounting for degenerate solutions of the energy loss are included in supplementary materials.

6 Experimental Evaluation

We evaluate NeuPSL's prediction performance and inference time on three tasks to demonstrate the significance of joint symbolic inference and learning. NeuPSL, implemented using the open-source PSL software package, can be integrated with any neural network library (here, we used TensorFlow). 2 Our investigation addresses the following questions: Q1) Can neuro-symbolic methods provide a boost over conventional purely data-driven neural models? Q2) Can we effectively leverage structural relationships across training examples through joint reasoning? Q3) How does NeuPSL compare with other neuro-symbolic methods in terms of time efficiency on large scale problems?

6.1 MNIST Addition

The first set of experiments are conducted on a variation of MNIST Addition, a widely used NeSy evaluation task [Manhaeve et al., 2018]. The task involves determining the sum of two lists of MNIST images. For example, a MNIST-Add1 addition is ([ ] + [ ] = 8), and a MNIST-Add2 addition is ([ , ] + [ , ] = 41). The challenge stems from the lack of labels for the MNIST images in the addition equation. Only the final sum of the equation is given, leaving the task of identifying the individual digits and determining their values up to the model being used.

While NeuPSL proves to be successful in the original MNIST-Add setting (appendix for further details), here we are interested in exploring the power of joint inference and learning capabilities in NeSy systems. We introduce a variant of the MNIST-Add task in which digits are reused across multiple addition examples, i.e., we introduce overlap. Figure 2 demonstrates the process of introducing overlap and how joint models narrow the space of possible labels when MNIST images are re-used. For instance, in the scenario presented in Figure 2, the same MNIST image of a zero is utilized in two separate additions. To comply with both addition constraints, the potential label space is restricted and can no longer include options such as two or three, as they would violate one of the addition rules. In contrast, a model performing independent reasoning would have no way of enforcing this constraint across examples.

In the overlap variant of MNIST-Add, we focus on lowdata settings to understand whether NeSy systems' joint reasoning can effectively leverage additional structure to overcome a lack of data. To create overlap, we begin with a set of n unique MNIST images from which we re-sample to create (n + m)/2 MNIST-Add1 and (n + m)/4 MNIST-Add2 additions. We vary the amount of overlap with m ∈ {0, n/2, n} and compare performance with n ∈ {40, 60, 80}. Results

2 Implementation details, hyperparameters, network architectures, hardware, and NeuPSL models, are described in the Appendix.

Code and Data: https://github.com/linqs/neupsl-ijcai23

Appendix: https://arxiv.org/abs/2205.14268

are reported over ten test sets of 1, 000 MNIST images with overlap proportional to the respective train set.

Figure 3 summarizes average performance for varying overlap settings. Each panel varies the number of additions for a set number of unique MNIST images. For example, the upper left panel presents the results obtained for MNISTAdd1 with 40 unique images used to generate 20, 30, and 40 additions. Initially, there is not enough structure from the additions with no overlap for symbolic inference to discern the correct digit labels for training the neural models. Then, despite the number of unique MNIST images remaining the same, as the number of additions increases, DPL and NeuPSL improve their prediction performance by leveraging the added joint information (Q2). In all cases, NeuPSL performs best and uses the added structure most efficiently. LTNs and the CNN baseline benefit the least from joint information, a consequence of both learning and inference being performed independently across batches of additions (Q1).

6.2 Visual Sudoku Classification

Inspired by the Visual Sudoku problem proposed by Wang et al. (2019), Augustine et al. (2022) introduced a novel NeSy task, Visual-Sudoku-Classification. In this task, 4x4 Sudoku puzzles are constructed using unlabeled MNIST images. The model must identify whether a puzzle is correct, i.e., no duplicate digits in any row, column, or square. Therefore this task does not require learning the underlying label for images but rather whether an entire puzzle is valid. For instance, [ ] does not need to belong to a "3" class, instead [ ] and [ ] need to be labeled as different symbols. Similar to MNIST-Add we explore an overlap variant in low-data settings, with overlapping MNIST images across puzzles.

We compare NeuPSL with two baselines, CNN-Visual and CNN-Digit. The first, CNN-Visual, takes the pixels for a Sudoku puzzle as input and outputs the probability the puzzle is valid. The second, CNN-Digit, is provided the (unfair) advantage of all sixteen image labels as input. We use this to verify whether a neural model can learn Sudoku rules. Scalably developing LTN and DPL models in this new setting is not straightforward due to the large dimensionality of the output space. A non-expert implementation of a visual sudoku model in DPL and LTN may result in suboptimal reports on model performance and are therefore not included.

Figure 4 shows the accuracy of NeuPSL and CNN models on Visual-Sudoku-Classification with varying amounts of overlap. CNN-Visual and CNN-Digit struggle to leverage the problem structure and fail to generalize even the highest data and overlap setting with 256 MNIST images across 64 puzzles. However, NeuPSL achieves 70% accuracy using roughly 64 MNIST images across 16 puzzles, again showing it efficiently leverages joint information across training examples (Q1 and Q2). This is a particularly impressive result as the neural network in the NeuPSL model was trained to be a 93% 4-digit distinguisher without digit labels.

6.3 Citation Network Node Classification

In our final experiment, we evaluate the performance of NeuPSL on two widely studied citation network node classification datasets: Citeseer and Cora [Sen et al., 2008]. In these datasets, symbolic models have the potential to improve predictions by leveraging the homophilic structure of the citation network, i.e., two papers connected in the network are more likely to have the same label. This setting differs from Visual-Sudoku-Classification and MNIST-Add as the symbolic relations are not always true. Moreover, the symbolic relations can be defined over a general and potentially large number of nodes in the network, i.e., a node can be connected to any number of neighbors.

We propose two NeuPSL models for citation network node classification. Both models integrate a neural network that uses a paper's features to provide an initial classification, which is then adjusted via symbolic reasoning. The first model, NeuPSLLP (Label Propagation), directly uses the bag-of-words feature vector, while the second model, NeuPSLLP+FP (Label + Feature Propagation), first performs the feature construction procedure as described in Wu et al. (2019) to obtain a richer representation to provide to the neural model. We examine the runtime and model performance of NeSy methods NeuPSLLP , NeuPSLLP+FP , DPL and its scalable extension, DeepStochLog [Winters et al., 2022], and a Graph Convolutional Network (GCN) [Kipf and Welling, 2017]. Additionally, we include the performance of two baselines, LPPSL and NeuralPSL. These baselines represent the distinct symbolic and neural components used in the NeuPSLLP model but perform only neural or symbolic reasoning, not both. We averaged the results over ten randomly sampled splits using 5% of the nodes for training, 5% of the nodes for validation, and 1000 nodes for testing.

Table 1 shows DeepStochLog, GCN, and NeuPSL all outperform the independent baselines (Q1), with NeuPSLLP+FP performing the best. These results demonstrate the power of using NeSy systems to effectively leverage structure to improve prediction performance. Additionally, NeuPSL is capable of scaling its joint inference process to larger structures, achieving higher accuracy with an 8 and 40 times speed up over DeepStochLog in Citeseer and Cora, respectively (Q3). Surprisingly, NeuPSL also achieves a higher prediction performance than even a GCN model while using significantly fewer trainable parameters.

7 Conclusion

In this paper, we introduced NeuPSL, a novel NeSy framework that integrates neural architectures and a tractable class of graphical models for jointly reasoning over symbolic relations and showed its utility across a range of neuro-symbolic tasks. There are many avenues for future work, including exploring different learning objectives, such as ones that balance traditional neural and energy-based losses and new application domains. Each of these is likely to provide new challenges and insights.

Acknowledgments

This work was partially supported by the National Science Foundation grant CCF-2023495 and a Google Faculty Research Award.

Contribution Statement

Connor Pryor and Charles Dickens contributed equally to this work.