Logic Tensor Networks for Semantic Image Interpretation

1 Introduction

Semantic Image Interpretation (SII) is the task of generating a structured semantic description of the content of an image. This structured description can be represented as a labelled directed graph. Each vertex corresponds to a bounding box of an object in the image, each edge represents a relation between pairs of objects. Vertices are labelled with a set of object types and edges are labelled with binary relations. Such a graph is also called a scene graph in [Krishna et al., 2016].

A major obstacle to be overcome by SII is the so-called semantic gap [Neumann and M¨oller, 2008], that is, the lack of a direct correspondence between low-level features of the

Artur d'Avila Garcez

City, University of London Northampton Square London EC1V 0HB, UK a.garcez@city.ac.uk

image and high-level semantic descriptions. To tackle this problem, a system for SII must learn the latent correlations that may exist between the numerical features that can be observed in an image and the semantic concepts associated with the objects. It is in this learning process that the availability of relational background knowledge can be of great help. Thus, recent SII systems have sought to combine, or even integrate, visual features obtained from data and symbolic knowledge in the form of logical axioms [Zhu et al., 2014; Chen et al., 2012; Donadello and Serafini, 2016].

The area of Statistical Relational Learning (SRL), or Statistical Artificial Intelligence (StarAI), seeks to combine datadriven learning, in the presence of uncertainty, with symbolic knowledge [Wang and Domingos, 2008; Bach et al., 2015; Gutmann et al., 2010; Diligenti et al., 2015; Rocktaschel et al., 2015; Ravkic et al., 2015]. However, only very few SRL systems have been applied to SII tasks (c.f. Section 2) due to the high complexity associated with image learning. Most systems for solving SII tasks have been based, instead, on deep learning and neural network models. These, on the other hand, do not in general offer a well-founded way of learning from data in the presence of relational logical constraints, requiring the neural models to be highly engineered from scratch. In this paper, we develop and apply for the first time, the SRL framework called Logic Tensor Networks (LTNs) to computationally challenging SII tasks. LTNs combine learning in deep networks with relational logical constraints [Serafini and d'Avila Garcez, 2016]. It uses a First-order Logic (FOL) syntax interpreted in the real numbers, which is implemented as a deep tensor network. Logical terms are interpreted as feature vectors in a real-valued n-dimensional space. Function symbols are interpreted as real-valued functions, and predicate symbols as fuzzy logic relations. This syntax and semantics, called real semantics, allow LTNs to learn efficiently in hybrid domains, where elements are composed of both numerical and relational information. We argue, therefore, that LTNs are a good candidate for learning SII because they can express relational knowledge in FOL which serves as constraints on the data-driven learning within tensor networks. Being LTN a logic, it provides a notion of logical consequence, which forms the basis for learning within LTNs, which is defined as best satisfiability, c.f. Section 4. Solving the best satisfiability problem amounts to finding the latent correlations that may exist between a relational background knowledge and numerical data attributes. This formulation enables the specification of learning as reasoning, a unique characteristic of LTNs, which is seen as highly relevant for SII. This paper specifies SII within LTNs, evaluating it on two important tasks: (i) the classification of bounding boxes, and (ii) the detection of the part-of relation between any two bounding boxes. Both tasks are evaluated using the PASCAL-PART dataset [Chen et al., 2014]. It is shown that LTNs improve the performance of the state-of-the-art object classifier Fast R-CNN [Girshick, 2015] on the bounding box classification task. LTNs also outperform a rule-based heuristic (which uses the inclusion ratio of two bounding boxes) in the detection of part-of relations between objects. Finally, LTNs are evaluated on their ability to handle errors, specifically misclassification of objects and part-of relations. Very large visual recognition datasets now exist which are noisy [Reed et al., 2014], and it is important for learning systems to become robust to noise. LTNs were trained systematically on progressively noisier datasets, with results on both SII tasks showing that LTN's logical constraints are capable of adding robustness to the system, in the presence of errors in the labels of the training data. The paper is organized as follows: Section 2 contrasts the LTN approach with related work which integrate visual features and background knowledge for SII. Section 3 specifies LTNs in the context of SII. Section 4 defines the best satisfiability problem in this context, which enables the use of LTNs for SII. Section 5 describes in detail the comparative evaluations of LTNs on the SII tasks. Section 6 concludes the paper and discusses directions for future work.

2 RelatedWork

The idea of exploiting logical background knowledge to improve SII tasks dates back to the early days of AI. In what follows, we review the most recent results in the area in comparison with LTNs.

Logic-based approaches have used Description Logics (DL), where the basic components of the scene are all assumed to have been already discovered (e.g. simple object types or spatial relations). Then, with logical reasoning, new facts can be derived in the scene from these basic components [Neumann and M¨oller, 2008; Peraldi et al., 2009]. Other logic-based approaches have used fuzzy DL to tackle uncertainty in the basic components [Hudelot et al., 2008; Dasiopoulou et al., 2009; Atif et al., 2014]. These approaches have limited themselves to spatial relations or to refining the labels of the objects detected. In [Donadello and Serafini, 2016], the scene interpretation is created by combining image features with constraints defined using DL, but the method is tailored to the part-of relation and cannot be extended easily to account for other relations. LTNs, on the other hand, should be able to handle any semantic relation. In [Marszalek and Schmid, 2007; Forestier et al., 2013], a symbolic Knowledge-base is used to improve object detection, but only the subsumption relation is explored and it is not possible to inject more complex knowledge using logical axioms.

A second group of approaches encodes background knowledge and visual features within probabilistic graphical models. In [Zhu et al., 2014; Nyga et al., 2014], visual features are combined with knowledge gathered from datasets, web resources or annotators, about object labels, properties (e.g., shape, colour, size) and affordances, using Markov Logic Networks (MLNs) [Richardson and Domingos, 2006] to predict facts in unseen images. Due to the specific knowledgebase schema adopted, the effectiveness of MLNs in this domain is evaluated only for Horn clauses, although the language of MLNs is more general. As a result, it is not easy to evaluate how the approach may perform with more complex axioms. In [Bach et al., 2015], a probabilistic fuzzy logic is used, but not with real semantics. Clauses are weighted and universally-quantified formulas are instantiated, as done by MLNs. This is different from LTNs where the universallyquantified formulas are computed by using an aggregation operation, which avoids the need for instantiating all variables.

In other related work, [Chen et al., 2012; Kulkarni et al., 2011] encode background knowledge into a generic Conditional Random Field (CRF), where the nodes represent detected objects and the edges represent logical relationships between objects. The task is to find a correct labelling for this graph. In [Chen et al., 2012], the edges encode logical constraints on a knowledge-base specified in DL. Although these ideas are close in spirit to the approach presented in this paper, they are not formalised as in LTNs, which use a deep tensor network and first-order logic, rather than CRFs or DL. In general, the logical theory behind the functions to be defined in the CRF is unclear. In [Kulkarni et al., 2011], potential functions are defined as text priors such as co-occurrence of terms found in the image descriptions of Flickr.

In a final group of approaches, here called language-priors, background knowledge is taken from linguistic models [Ramanathan et al., 2015; Lu et al., 2016]. In [Ramanathan et al., 2015], a neural network is built integrating visual features and a linguistic model to predict semantic relationships between bounding boxes. The linguistic model is a set of rules derived fromWORDNET [Fellbaum, 1998], stating which types of semantic relationships occur between a subject and an object. In [Lu et al., 2016], a similar neural network is proposed for the same task but with a more sophisticated language model, embedding in the same vector space triples of the form subject-relation-object, such that semantically similar triples are mapped closely together in the embedding space. In this way, even if no examples exist of some triples in the data, the relations can be inferred from similarity to more frequent triples. A drawback, however, is the possibility of inferring inconsistent triples, such as e.g. man-eats-chair, due to the embedding. LTNs avoid this problem with a logic-based approach (in the above example, with an axiom to the effect that chairs are not normally edible). LTNs can also handle exceptions, offering a system capable of dealing with crisp axioms and real-valued data, as specified in what follows.

3 Logic Tensor Networks

Let L be a first-order logic language, whose signature is composed of three disjoint sets C, F and P, denoting constants, functions and predicate symbols, respectively. For any function or predicate symbol s, let α(s) denote its arity. Logical formulas in L allow one to specify relational knowledge, e.g. the atomic formula partOf(o1, o2), stating that object o1 is a part of object o2, the formulae ∀xy(partOf(x, y) → ¬partOf(y, x)), stating that the relation partOf is asymmetric, or ∀x(Cat(x) → ∃y(partOf(x, y)∧Tail(y))), stating that every cat should have a tail. In addition, exceptions are handled by allowing formulas to be interpreted in fuzzy logic, such that in the presence of an example of, say, a tailless cat, the above formula can be interpreted naturally as normally, every cat has a tail; this will be exemplified later.

Semantics of L: We define the interpretation domain as a subset of Rn , i.e. every object in the domain is associated with a n-dimensional vector of real numbers. Intuitively, this n-tuple represents n numerical features of an object, e.g. in the case of a person, their name in ASCII, height, weight, social security number, etc. Functions are interpreted as realvalued functions, and predicates are interpreted as fuzzy relations on real vectors. To emphasise the fact that we interpret symbols as real numbers, we use the term grounding instead of interpretation 1 in the following definition of semantics.

Definition 1 Let n ∈ N. An n-grounding, or simply grounding, G for a FOL L is a function defined on the signature of L satisfying the following conditions:

1. G(c) ∈ Rn for every constant symbol c ∈ C;
2. G(f) ∈ Rn·α(f) -→Rn for every f ∈ F;
3. G(P) ∈ Rn·α(P) -→ [0, 1] for every P ∈ P.

Given a grounding G, the semantics of closed terms and atomic formulas is defined as follows:

The semantics for connectives is defined according to fuzzy logic; using for instance the Lukasiewicz t-norm: 2

The LTN semantics for ∀ is defined in [Serafini and d'Avila Garcez, 2016] using the min operator, that is, G(∀xφ(x)) = mint∈term(L) G(φ(t))), where term(L) is the set of instantiated terms of L. This, however, is inadequate for our purposes as it does not tolerate exceptions (the presence of a single exception to the universally-quantified formulae, such as e.g. a cat without a tail, would falsify the formulae). Instead, our aim in SII is that the more examples that satisfy a formulae φ(x), the higher the truth-value of ∀xφ(x) should be. To capture this, we use for the semantics of ∀ a mean-operator:

1 In logic, the term grounding indicates the operation of replacing the variables of a term or formula with constants or terms that do not contain other variables. To avoid any confusion, we use the synonym instantiation for this purpose. It is worth noting that in LTN, differently fromMLNs, the instantiation of every first order formula is not required.

2 Examples of t-norms include Lukasiewicz, product and G¨odel. The Lukasiewicz t-norm is µLuk(x, y) = max(0, x+y-1), product t-norm is µPr(x, y) = x · y, and G¨odel t-norm is µmax(x, y) = min(x, y). See [Bergmann, 2008] for details.

Finally, the classical semantics of ∃ is uniquely determined by the semantics of ∀, by making ∃ equivalent to ¬∀¬. This approach, however, has a drawback too when it comes to SII: if we adopt, for instance, the arithmetic mean for the semantic of ∀ then G(∀xφ(x)) = G(∃xφ(x)). Therefore, we shall interpret existential quantification via Skolemization: every formula of the form ∀x1, . . . , xn(. . . ∃yφ(x1, . . . , xn, y)) is rewritten as ∀x1, . . . , xn(. . . φ(x1, . . . , xn, f(x1, . . . , xn))), by introducing a new n-ary function symbol, called Skolem function. In this way, existential quantifiers can be eliminated from the language by introducing Skolem functions.

Formalizing SII in LTNs: To specify the SII problem, as defined in the introduction, we consider a signature ΣSII = 〈C,F,P〉, where C = ⋃p∈Pics b(p) is the set of identifiers for all the bounding boxes in all the images, F = ∅, and P = {P1,P2}, where P1 is a set of unary predicates, one for each object type, e.g. P1 = {Dog, Cat, Tail,Muzzle,Train, Coach, . . . }, and P2 is a set of binary predicates representing relations between objects. Since in our experiments we focus on the part-of relation, P2 = {partOf}. The FOL formulas based on this signature can specify (i) simple facts, e.g. the fact that bounding box b contains a cat, written Cat(b), the fact that b contains either a cat or a dog, written Cat(b) ∨ Dog(b), etc., and (ii) general rules such as ∀x(Cat(x) → ∃y(partOf(x, y) ∧ Tail(y))).

A grounding for ΣSII can be defined as follows: each constant b, denoting a bounding box, can be associated with a set of geometric features and a set of semantic features computed with a bounding box detector. Specifically, each bounding box is associated with geometric features describing the position and the dimension of the bounding box, and semantic features describing the classification score returned by the bounding box detector for each class. For example, for each bounding box b ∈ C, Ci ∈ P1, G(b) is the R4+|P1| vector:

where the last four elements are the coordinates of the top-left and bottom-right corners of b, and class(Ci, b) ∈ [0, 1] is the classification score of the bounding box detector for b.

An example of groundings for predicates can be defined by taking a one-vs-all multi-classifier approach, as follows. First, define the following grounding for each class Ci ∈ P1 (below, x = 〈x1, . . . , x|P1|+4 〉 is the vector corresponding to the grounding of a bounding box):

Then, a simple rule-based approach for defining a grounding for the partOf relation is based on the na¨ıve assumption that the more a bounding box b is contained within a bounding box b′ , the higher the probability should be that b is part of b′ . Accordingly, one can define G(partOf(b, b′ )) as the inclusion ratio ir(b, b′ ) of bounding box b, with grounding x, into bounding box b′ , with grounding x′ (formally, ir(b, b′ ) = area(b∩b′ ) area(b) ). A slightly more sophisticated rulebased grounding for partOf (used as baseline in the experiments to follow) takes into account also type compatibilities by multiplying the inclusion ratio by a factor wij . Hence, we define G(partOf(b, b′ )) as follows:

3 The popular mean operators, arithmetic, geometric and harmonic mean, are obtained by setting p = 1, 2, and -1, respectively.

for some threshold thir (we use thir > 0.5), and with wij = 1 if Ci is a part of Cj, and 0 otherwise. Given the above grounding, we can compute the grounding of any atomic formula, e.g. Cat(b1), Dog(b2), leg(b3), partOf(b3, b1), partOf(b3, b2), thus expressing the degree of truth of the formula. The rule-based groundings (Eqs. (1) and (2)) may not satisfy all the constraints to be imposed. E.g., the classification score may be wrong, a bounding box may include another one which is not in the part-of relation, etc. Furthermore, in many situations it is not possible to define a grounding a priori. Instead, groundings should be learned automatically from data, by optimizing the truth-values of the formulas in the background knowledge. This is discussed next.

4 Learning as Best Satisfiability

A partial grounding, denoted by ˆ G, is a grounding that is defined on a subset of the signature of L. A grounding G is said to be a completion of ˆ G, if G is a grounding for L and coincides with ˆ G on the symbols where ˆ G is defined.

Definition 2 A grounded theory GT is a pair 〈K, ˆ G〉 with a set K of closed formulas and a partial grounding ˆ G.

Definition 3 A grounding G satisfies a GT 〈K, ˆ G〉 if G completes ˆ G and G(φ) = 1 for all φ ∈ K. A GT 〈K, ˆ G〉 is satisfiable if there exists a grounding G that satisfies 〈K, ˆ G〉.

According to the previous definition, deciding the satisfiability of 〈K, ˆ G〉 amounts to searching for a grounding ˆ G such that all the formulas of K are mapped to 1. Differently from the classical satisfiability, when a GT is not satisfiable, we are interested in the best possible satisfaction that we can reach with a grounding. This is defined as follows.

Definition 4 Let 〈K, ˆ G〉 be a grounded theory. We define the best satisfiability problem as the problem of finding a grounding G ∗ that maximizes the truth-values of the conjunction of all clauses cl ∈ K, i.e. G ∗ = argmax ˆ G⊆G∈G G(∧cl∈K cl).

Grounding G ∗ captures the latent correlation between quantitative attributes of objects and their categorical/relational properties. Not all functions are suitable as a grounding; they should preserve some form of regularity. If G(Cat)(x) ≈ 1 (the bounding box with feature vector x contains a cat) then for every x′ close to x (i.e. for every bounding box with features similar to x), one should have G(Cat)(x′ ) ≈ 1. In particular, we consider groundings of the following form.

Function symbols are grounded to linear transformations. If f is a m-ary function symbol, then G(f) is of the form:

where v = 〈vᵀ 1 , . . . , vᵀm〉ᵀ is the mn-ary vector obtained by concatenating each vi. The parameters for G(f) are the n × mn real matrix Mf and the n-vector Nf .

The grounding of an m-ary predicate P, namely G(P), is defined as a generalization of the neural tensor network (which has been shown effective at knowledge completion in the presence of simple logical constraints [Socher et al., 2013]), as a function from Rmn to [0, 1], as follows:

with σ the sigmoid function. The parameters for P are: W [1:k] P , a 3-D tensor in Rk×mn×mn , VP ∈ Rk×mn , bP ∈ Rk and uP ∈ Rk . This last parameter performs a linear combination of the quadratic features given by the tensor product. With this encoding, the grounding (i.e. truth-value) of a clause can be determined by a neural network which first computes the grounding of the literals contained in the clause, and then combines them using the specific t-norm.

In what follows, we describe how a suitable GT can be built for SII. Let Pics t ⊆ Pics be a set of bounding boxes of images correctly labelled with the classes that they belong to, and let each pair of bounding boxes be correctly labelled with the part-of relation. In machine learning terminology, Pics t is a training set without noise. In real semantics, a training set can be represented by a theory Texpl = 〈Kexpl, ˆ G〉, where Kexpl contains the set of closed literals Ci(b) (resp. ¬Ci(b)) and partOf(b, b′ ) (resp. ¬partOf(b, b′ )), for every bounding box b labelled (resp. not labelled) with Ci and for every pair of bounding boxes 〈b, b′ 〉 connected (resp¬partOf(b, b′ ). not connected) by the partOf relation. The partial grounding ˆ G is defined on all bounding boxes of all the images in Pics where both the semantic features class(Ci, b) and the bounding box coordinates are computed by the Fast R-CNN object detector [Girshick, 2015]. ˆ G is not defined for the predicate symbols in P and is to be learned. Texpl contains only assertional information about specific bounding boxes. This is the classical setting of machine learning where classifiers (i.e. the grounding of predicates) are inductively learned from positive examples (such as partOf(b, b′ )) and negative examples (¬partOf(b, b′ )) of a classification. In this learning setting, mereological constraints such as "cats have no wheels" or "a tail is a part of a cat" are not taken into account. Examples of mereological constraints state, for instance, that the part-of relation is asymmetric (∀xy(partOf(x, y) → ¬partOf(y, x))), or lists the several parts of an object (e.g. ∀xy(Cat(x) ∧ partOf(x, y) → Tail(y) ∨ Muzzle(y))), or even, for simplicity, that every whole object cannot be part of another object (e.g. ∀xy(Cat(x) → ¬partOf(x, y))) and every part object cannot be divided further into parts (e.g. ∀xy(Tail(x) → ¬partOf(y, x))). This general knowledge is available from on-line resources, such as WORDNET [Fellbaum, 1998], and can be retrieved by inheriting the meronymy relations for every concept corresponding to a whole object. A grounded theory that considers also mereological constraints as prior knowledge can be constructed by adding such axioms to Kexpl. More formally, we define Tprior = 〈Kprior, ˆ G〉, where Kprior = Kexpl +M, and M is the set of mereological axioms. To check the role of M, we evaluate both theories and then compare results.

5 Experimental Evaluation

We evaluate the performance of our approach for SII 4 on two tasks, namely, the classification of bounding boxes and the detection of partOf relations between pairs of bounding boxes. We chose the part-of relation because both data (the PASCAL-PART-dataset [Chen et al., 2014]) and ontologies (WORDNET) are available on the part-of relation. In addition, part-of can be used to represent, via reification, a large class of relations [Guarino and Guizzardi, 2016] (e.g., the relation "a plant is lying on the table" can be reified in an object of type "lying event" whose parts are the plant and the table). However, it is worth noting that many other relations could have been included in this evaluation. The time complexity of LTN grows linearly with the number of axioms.

We also evaluate the robustness of our approach with respect to noisy data. It has been acknowledged by many that, with the vast growth in size of the training sets for visual recognition [Krishna et al., 2016], many data annotations may be affected by noise such as missing or erroneous labels, non-localised objects, and disagreements between annotations, e.g. human annotators often mistake "part-of" for the "have" relation [Reed et al., 2014].

We use the PASCAL-PART-dataset that contains 10103 images with bounding boxes annotated with object-types and the part-of relation defined between pairs of bounding boxes. Labels are divided into three main groups: animals, vehicles and indoor objects, with their corresponding parts and "partof" label. Whole objects inside the same group can share parts. Whole objects of different groups do not share any parts. Labels for parts are very specific, e.g. "left lower leg". Thus, without loss of generality, we have merged the bounding boxes that referred to the same part into a single bounding box, e.g. bounding boxes labelled with "left lower leg" and "left upper leg" were merged into a single bounding box of type "leg". In this way, we have limited our experiments to a dataset with 20 labels for whole objects and 39 labels for parts. In addition, we have removed from the dataset any bounding boxes with height or width smaller than 6 pixels. The images were then split into a training set with 80%, and a test set with 20% of the images, maintaining the same proportion of the number of bounding boxes for each label.

Object Type Classification and Detection of the PartOf Relation: Given a set of bounding boxes detected by an object detector (we use Fast-RCNN), the task of object classification is to assign to each bounding box an object type. The task of Part-Of detection is to decide, given two bounding boxes, if the object contained in the first is a part of the object contained in the second. We use LTN to resolve both tasks simultaneously. This is important because a bounding box type and the part-of relation are not independent. Their dependencies are specified in LTN using background knowledge in the form of logical axioms.

4 LTN has been implemented as a Google TENSORFLOWTM library. Code, partOf ontology, and dataset are available at https://gitlab.fbk.eu/donadello/LTN_IJCAI17

To show the effect of the logical axioms, we train two LTNs: the first containing only training examples of object types and part-of relations (Texpl), and the second containing also logical axioms about types and part-of (Tprior). The LTNs were set up with tensor of k = 6 layers and a regularization parameter λ = 10-10 . We chose Lukasiewicz's T-norm (µ(a, b) = max(0, a + b - 1)) and use the harmonic mean as aggregation operator. We ran 1000 training epochs of the RMSProp learning algorithm available in TENSORFLOWTM . We compare results with the Fast RCNN at object type classification (Eq.(1)), and the inclusion ratio ir baseline (Eq.(2)) at the part-of detection task. 5 If ir is larger than a given threshold th (in our experiments, th = 0.7) then the bounding boxes are said to be in the partOf relation. Every bounding box b is classified into C ∈ P1 if G(C(b)) ≥ th. With this, a bounding box can be classified into more than one class. For each class, precision and recall are calculated in the usual way. Results for indoor objects are shown in Figure 1 where AUC is the area under the precision-recall curve. The results show that, for both object types and the part-of relation, the LTN trained with prior knowledge given by mereological axioms has better performance than the LTN trained with examples only. Moreover, prior knowledge allows LTN to improve the performance of the Fast R-CNN (FRCNN) object detector. Notice that the LTN is trained using the Fast R-CNN results as features. FRCNN assigns a bounding box to a class if the values of the corresponding semantic features exceed th. This is local to the specific semantic features. If such local features are very discriminative (which is the case in our experiments) then very good levels of precision can be achieved. Differently from FRCNN, LTNs make a global choice which takes into consideration all (semantic and geometric) features together. This should offer robustness to the LTN classifier at the price of a drop in precision. The logical axioms compensate this drop. For the other object types (animals and vehicles), LTN has results comparable to FRCNN: FRCNN beats Tprior by 0.05 and 0.037 AUC, respectively, for animals and vehicles. Finally, we have performed an initial experiment on small data, on the assumption that the LTN axioms should be able to compensate a reduction in training data. By removing 50% of the training data for indoor objects, a similar performance to Tprior with the full training set can be achieved: 0.767 AUC for object types and 0.623 AUC for the part-of relation, which shows an improvement in performance.

Robustness to Noisy Training Data: In this evaluation, we show that logical axioms improve the robustness of LTNs in the presence of errors in the labels of the training data. We have added an increasing amount of noise to the PASCALPART-dataset training data, and measured how performance degrades in the presence and absence of axioms. For k ∈ {10, 20, 30, 40}, we randomly select k% of the bounding boxes in the training data, and randomly change their classification labels. In addition, we randomly select k% of pairs of bounding boxes, and flip the value of the part-of relation's label. For each value of k, we train LTNs T k expl and T k prior and evaluate results on both SII tasks as done before. As expected, adding too much noise to training labels leads to a large drop in performance. Figure 2 shows the AUC measures for indoor objects with increasing error k. Each pair of bars indicates the AUC of T k prior , T k expl , for a given k% of errors. Results indicate that the LTN axioms offer robustness to noise: in addition to the expected overall drop in performance, an increasing gap can be seen between the drop in performance of the LTN trained with examples only and the LTN trained including background knowledge.

5 A direct comparison with [Chen et al., 2012] is not possible because their code was not available.

(b) LTNs with prior knowledge outperform the rulebased approach of Eq.2 in the detection of part-of relations, achieving AUC of 0.598 in comparison with 0.172.

Figure 1: Precision-recall curves for indoor objects type classification and the partOf relation between objects.

6 Conclusion and FutureWork

SII systems are required to address the semantic gap problem: combining visual low-level features with high-level concepts. We argue that the problem can be addressed by the integration of numerical and logical representations in deep learning. LTNs learn from numerical data and logical constraints, enabling approximate reasoning on unseen data to predict new facts. In this paper, LTNs were shown to improve on state-ofthe-art method Fast R-CNN for bounding box classification, and to outperform a rule-based method at learning part-of relations in the PASCAL-PART-dataset. Moreover, LTNs were evaluated on how to handle noisy data through the systematic creation of training sets with errors in the labels. Results indicate that relational knowledge can add robustness to neural systems. As future work, we shall apply LTNs to larger datasets such as VISUAL GENOME, and continue to compare the various instances of LTN with SRL, deep learning and other neural-symbolic approaches on such challenging visual intelligence tasks.