BERT-INT:A BERT-based Interaction Model For Knowledge Graph Alignment

1 Introduction

DBpedia, Freebase, YAGO and so on have been published as noteworthy large and freely available knowledge graphs (KGs), which can benefit many applications such as question answering and recommendation [Tong et al., 2019]. However, a single KG is far from complete to support such applications with sufficient facts, which demands an effective way to align entities across KGs. To solve the problem, much attention has been paid to leveraging the graph structures to align entities. The specific technique has evolved from the traditional KG embedding models such as MTransE [Chen et al., 2017] and IPTransE [Zhu et al., 2017] to recent emergent graph neural networks such as attention-based GCN [Xu et al., 2019], highway GCN [Wu et al., 2019b], relation-aware GCN [Wu et al., 2019a] and VR-GCN [Ye et al., 2019].

Despite much effort taken on graph structures, the side information of entities such as name, description and attributes may play a more important role on many tasks on KGs, because: (1) the KGs are usually sparse, with a large number of long-tail entities whose structural embeddings have low expressiveness [Guo et al., 2019]. For example, in DBpediaZH 1 , the long-tail entities that appear less than 5 times occupy 74.1%; and (2) existing works have demonstrated the effect of side information on several tasks. For example, HMAN [Yang et al., 2019] presents that on the task of entity alignment, when ignoring entity descriptions, HitRatio@1 will drop at least 30%.

∗Contact Author

To leverage the side information to align KGs, the most popular way is to initialize the embedding of each node by its side information and apply variant GCN models to update a node embedding by aggregating all neighbors' embeddings [Wang et al., 2018; Wu et al., 2019a; Xu et al., 2019]. However, since different KGs are highly heterogeneous, it is not always the case that equivalent entities share similar neighbors. Taking an example from HMAN in Figure 1, the nodes in rectangles are entities to be aligned and the nodes in ovals are the neighbors. We can see that for the neighbor "English" in G2, no counterpart can be found from the neighbors in G1. In this case, propagating the dissimilar neighbors in GCNs, especially the hub entities such as the neighbor "English" with 832 relations, may introduce noises and thus harm performance [Yang et al., 2019]. Although some works distinguish the influence from different neighbors [Cao et al., 2019; Wu et al., 2019a; Xu et al., 2019], essentially, the GCNlike models still mix the side information of all the neighbors to represent an entity. HMAN has been aware of the issue, thus it thoroughly separates the modeling process of side information and graph structures. However, it discards the side information of neighbors. Moreover, similar to aggregating neighbors, it aggregates all the attributes together to represent an entity like most of the works did [Sun et al., 2017; Zhang et al., 2019], which also results in noisy matches between entities.

To deal with the noisy matches caused by aggregating neighbors or attributes, we propose BERT-INT-i.e., a BERT-based INTeraction model to only leverage the side information, especially the names/descriptions of current entities and neighbors, and attributes of current entities in a unified way. Generally, we mimic the cognitive process of comparing two entities by humans, who usually first compare current entities and then continue to check whether there are similar neighbors. Following this, based on the embedding of any piece of name, description and attribute, we compare each pair of neighbors or attributes, referred to as interactions henceforth, instead of aggregating them. In this way, we can capture the fine-grained exact/semantic matches between neighbors and get rid of the negative influence from the dissimilar neighbors, as presented on the right of Figure 1.

1 https://wiki.dbpedia.org/downloads-2016-10

Although we leverage the side information of neighbors, unlike the variant GCN models, the structural characteristics of KGs are totally ignored in our model, because we never propagate the information of neighbors in graphs. From this point of view, we claim that BERT-INT only leverages side information. Just because of this, BERT-INT is capable of inductive learning, i.e., we can train BERT-INT on two aligned KGs and apply it to predict the matches between unseen entities in other KGs. However, if adopting the variant GCN models, the test entities should be included in the training KGs. In summary, we propose an inductive learning-based model to comprehensively leverage the name/description-view interaction, neighbor-view interactions and attribute-view interactions to align entities. By experiments, we demonstrate that BERT-INT significantly outperforms the state-of-the-art models by 1.9-9.7% in HitRatio@1.

2 ProblemDefinition

Definition 1. Knowledge Graph: We denote a KG as G = (E, R, A, V ), where each e ∈ E, r ∈ R, a ∈ A and v ∈ V denotes an entity, a relation, an attribute and a value respectively. N τ (e) = {(ri, ei)}|N τ (e)| i=1 denotes the set of all the τhop neighbors of entity e with τ as the number of hops, where each of the i-th neighbor contains a neighboring relation ri and the corresponding entity ei. A(e) = {(ai, vi)}|A(e)| i=1 denotes the set of the attributes of e, where each of the i-th attribute contains a name ai and the corresponding value vi. N(e) without τ indicates all the neighbors of e. |N τ (e)| and |A(e)| is the number of N τ (e) and A(e) respectively.

Problem 1. Knowledge Graph Alignment: Given two KGs G and G′ and a set of already aligned entity pairs I = {(e ∼ e′ )}, we aim at learning a ranking function f : E × E′ → R to calculate a similarity score between two entities, based on which we rank the correctly aligned entity e′ to any queried entity e as high as possible among all the entities in E′ .

3 BERT-INTModel

This section introduces the proposed BERT-INT, which consists of a BERT model that is used as a basic representation unit to embed the name, description, attribute and value of an entity, and an interaction model built upon the BERT embeddings to compute the interactions between these embeddings. The interactions are further divided into the name/description-view interaction, the neighbor-view interactions, and the attribute-view interactions. A unified dual aggregation function is applied to extract features from the neighbor-view and attribute-view interactions to estimate the matching score of entities. Besides, to build comprehensive neighbor-view interactions, the interactions between the neighboring entities, the corresponding neighboring relations, and the multi-hop neighbors are also considered. Figure 2 shows the whole framework.

3.1 Basic BERT Unit

We treat entity alignment as the downstream objective to finetune a pre-trained BERT model. Specifically, we first construct the training data D = {(e, e′+, e′-)}, where each triplet (e, e′+, e′-) ∈ D contains a queried entity e ∈ E, the rightly aligned counterpart e′+ ∈ E′ and a negative counterpart e′- randomly sampled from E′ . For each entity e in the dataset, we apply a pre-trained multi-lingual BERT 2 to accept its name/description as the input, filter the CLS embedding of BERT by a MLP layer to obtain

and use a pairwise margin loss to fine-tune BERT:

where m is a margin enforced between the positive pairs and negative pairs, and g(e, e′ ) is instantiated as l1 distance to measure the similarity between C(e) and C(e′ ). Negative pairs are sampled according to the cosine similarity of two entities [Sun et al., 2018].

2 https://github.com/google-research/bert

For the input of BERT, we give priority to the description as it contains richer information. We use the name when the description is missing. Different from HMAN that directly uses the embeddings of the descriptions to align entities [Yang et al., 2019], we use these embeddings as basic units to compose the following interaction model. Note we can also connect the basic BERT unit with the following interaction model as an end-to-end model to fine-tune BERT and train the interaction model simultaneously. In practice, considering the GPU memory and the running efficiency, we fine-tune the basic BERT unit beforehand and freeze its parameters in the following interaction model.

3.2 BERT-based InteractionModel

Based on the basic BERT unit, we build the interaction model which consists of the name/description-view interaction, the neighbor-view and the attribute-view interactions.

Name/Description-view Interaction

We apply the basic BERT unit on the names/descriptions of e and e′ to obtain C(e) and C(e′ ), and then calculate their cosine similarity as the name/description-view interaction.

Neighbor-view Interactions

We build the interactions between the neighbors N(e) and N(e′ ). The general idea is to compare the names/descriptions of each neighbor pair rather than learning a global representation for e or e′ by aggregating the names/descriptions of all their neighbors as existing works did [Wu et al., 2019a; Xu et al., 2019]. This similar idea is widely used in information retrieval to capture the exact and soft matches between a query and a candidate document [Xiong et al., 2017]. Specifically, we apply the basic BERT unit to obtain {C(ei)}|N(e)| i=1 and {C(e′i )}|N(e′ )| i=1 for e and e′ 's neighboring entities based on their names/descriptions, compute a similarity matrix between the two embedding sets and then apply a dual aggregation function to extract the similarity features from the matrix. We use S to represent the interactions between the neighbors of e and e′ , with each element sij standing for the interaction, i.e., the cosine similarity sij = C(ei)·C(e′j ) ∥C(ei)∥·∥C(e′j )∥ between C(ei) and C(e′j ), where C(ei) and C(e′j ) are obtained by Eq.(1) for the i-th neighbor of e and the j-th neighbor of e′ respectively.

We apply a dual aggregation function to extract the similarity features along both the rows and columns of S. CNN and RNN are usually used as the aggregation function to extract the matching patterns from the similarity matrix between two sentences. Different from sentences, the neighbors are disordered and independent from each other. Thus we adopt a RBF kernel aggregation function [Xiong et al., 2017] to extract features about the accumulation of similarities.

Before making use of the RBF kernel aggregations, we first apply a max pooling operation on each row Si = {si0, · · · , sij, · · · , sin} to get the maximal similarity smax i , i.e., we select the most possibly aligned counterpart from the neighbors of e′ for the i-th neighbor of e. This is because, following the one-one mapping assumption, we only care about how similar the most possibly aligned counterpart

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

D

8

M

0

u

v

2

U

7

o

X

j

O

r

N

k

S

E

+

q

>

A

C

B

c

V

L

F

J

3

d

Z

H

Q

P

G

Y

T

f

z

y

I

W

/

n

p

g

m

K

R

w

5

9

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

D

8

M

0

u

v

2

U

7

o

X

j

O

r

N

k

S

E

+

q

>

A

C

B

c

V

L

F

J

3

d

Z

H

Q

P

G

Y

T

f

z

y

I

W

/

n

p

g

m

K

R

w

5

9

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

D

8

M

0

u

v

2

U

7

o

X

j

O

r

N

k

S

E

+

q

>

A

C

B

c

V

L

F

J

3

d

Z

H

Q

P

G

Y

T

f

z

y

I

W

/

n

p

g

m

K

R

w

5

9

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

D

8

M

0

u

v

2

U

7

o

X

j

O

r

N

k

S

E

+

q

>

A

C

B

c

V

L

F

J

3

d

Z

H

Q

P

G

Y

T

f

z

y

I

W

/

n

p

g

m

K

R

w

5

9

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

R

q

H

w

y

I

V

u

X

U

/

B

g

F

K

A

d

5

f

>

7

3

c

N

S

8

E

J

r

9

L

C

p

Q

Y

2

o

+

z

D

m

G

Z

j

v

O

0

k

M

W

P

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

R

q

H

w

y

I

V

u

X

U

/

B

g

F

K

A

d

5

f

>

7

3

c

N

S

8

E

J

r

9

L

C

p

Q

Y

2

o

+

z

D

m

G

Z

j

v

O

0

k

M

W

P

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

R

q

H

w

y

I

V

u

X

U

/

B

g

F

K

A

d

5

f

>

7

3

c

N

S

8

E

J

r

9

L

C

p

Q

Y

2

o

+

z

D

m

G

Z

j

v

O

0

k

M

W

P

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

R

q

H

w

y

I

V

u

X

U

/

B

g

F

K

A

d

5

f

>

7

3

c

N

S

8

E

J

r

9

L

C

p

Q

Y

2

o

+

z

D

m

G

Z

j

v

O

0

k

M

W

P

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

B

R

V

+

m

W

d

3

E

0

G

/

T

8

A

N

M

>

7

c

S

J

U

r

q

9

L

C

p

5

I

Q

Y

F

2

o

z

u

D

H

Z

j

w

y

f

g

v

O

X

k

P

K

n

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

B

R

V

+

m

W

d

3

E

0

G

/

T

8

A

N

M

>

7

c

S

J

U

r

q

9

L

C

p

5

I

Q

Y

F

2

o

z

u

D

H

Z

j

w

y

f

g

v

O

X

k

P

K

n

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

B

R

V

+

m

W

d

3

E

0

G

/

T

8

A

N

M

>

7

c

S

J

U

r

q

9

L

C

p

5

I

Q

Y

F

2

o

z

u

D

H

Z

j

w

y

f

g

v

O

X

k

P

K

n

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

B

R

V

+

m

W

d

3

E

0

G

/

T

8

A

N

M

>

7

c

S

J

U

r

q

9

L

C

p

5

I

Q

Y

F

2

o

z

u

D

H

Z

j

w

y

f

g

v

O

X

k

P

K

n

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

u

N

A

E

r

O

5

C

f

o

J

8

K

P

/

d

p

m

Y

>

B

7

3

c

V

S

w

M

2

X

+

T

I

g

L

j

W

Q

Z

v

R

z

9

F

D

q

H

0

y

k

n

U

G

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

u

N

A

E

r

O

5

C

f

o

J

8

K

P

/

d

p

m

Y

>

B

7

3

c

V

S

w

M

2

X

+

T

I

g

L

j

W

Q

Z

v

R

z

9

F

D

q

H

0

y

k

n

U

G

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

u

N

A

E

r

O

5

C

f

o

J

8

K

P

/

d

p

m

Y

>

B

7

3

c

V

S

w

M

2

X

+

T

I

g

L

j

W

Q

Z

v

R

z

9

F

D

q

H

0

y

k

n

U

G

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

u

N

A

E

r

O

5

C

f

o

J

8

K

P

/

d

p

m

Y

>

B

7

3

c

V

S

w

M

2

X

+

T

I

g

L

j

W

Q

Z

v

R

z

9

F

D

q

H

0

y

k

n

U

G

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

8

7

d

M

u

z

B

X

+

F

O

c

p

S

W

E

U

I

o

>

A

3

V

N

J

r

q

/

9

L

C

5

Q

Y

2

T

D

H

m

G

Z

j

w

y

n

P

f

v

g

K

R

k

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

8

7

d

M

u

z

B

X

+

F

O

c

p

S

W

E

U

I

o

>

A

3

V

N

J

r

q

/

9

L

C

5

Q

Y

2

T

D

H

m

G

Z

j

w

y

n

P

f

v

g

K

R

k

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

8

7

d

M

u

z

B

X

+

F

O

c

p

S

W

E

U

I

o

>

A

3

V

N

J

r

q

/

9

L

C

5

Q

Y

2

T

D

H

m

G

Z

j

w

y

n

P

f

v

g

K

R

k

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

8

7

d

M

u

z

B

X

+

F

O

c

p

S

W

E

U

I

o

>

A

3

V

N

J

r

q

/

9

L

C

5

Q

Y

2

T

D

H

m

G

Z

j

w

y

n

P

f

v

g

K

R

k

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

N

/

W

f

9

3

m

J

p

8

Q

O

d

C

Y

g

K

M

>

A

E

X

c

V

D

L

S

F

U

2

v

q

k

o

u

H

w

7

n

B

j

5

+

Z

G

r

z

R

I

T

y

P

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

N

/

W

f

9

3

m

J

p

8

Q

O

d

C

Y

g

K

M

>

A

E

X

c

V

D

L

S

F

U

2

v

q

k

o

u

H

w

7

n

B

j

5

+

Z

G

r

z

R

I

T

y

P

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

N

/

W

f

9

3

m

J

p

8

Q

O

d

C

Y

g

K

M

>

A

E

X

c

V

D

L

S

F

U

2

v

q

k

o

u

H

w

7

n

B

j

5

+

Z

G

r

z

R

I

T

y

P

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

N

/

W

f

9

3

m

J

p

8

Q

O

d

C

Y

g

K

M

>

A

E

X

c

V

D

L

S

F

U

2

v

q

k

o

u

H

w

7

n

B

j

5

+

Z

G

r

z

R

I

T

y

P

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

X

V

w

I

W

y

Q

N

j

d

5

k

f

m

0

>

A

B

7

3

c

S

E

J

U

r

q

/

9

L

C

p

Y

F

2

o

+

z

u

D

H

G

Z

g

v

O

R

M

P

K

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

X

V

w

I

W

y

Q

N

j

d

5

k

f

m

0

>

A

B

7

3

c

S

E

J

U

r

q

/

9

L

C

p

Y

F

2

o

+

z

u

D

H

G

Z

g

v

O

R

M

P

K

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

X

V

w

I

W

y

Q

N

j

d

5

k

f

m

0

>

A

B

7

3

c

S

E

J

U

r

q

/

9

L

C

p

Y

F

2

o

+

z

u

D

H

G

Z

g

v

O

R

M

P

K

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

X

V

w

I

W

y

Q

N

j

d

5

k

f

m

0

>

A

B

7

3

c

S

E

J

U

r

q

/

9

L

C

p

Y

F

2

o

+

z

u

D

H

G

Z

g

v

O

R

M

P

K

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

E

V

X

S

L

j

B

w

Y

M

o

v

Q

Z

u

d

O

U

>

A

7

3

c

N

J

2

+

T

C

p

I

g

W

/

m

R

P

z

9

5

8

r

n

f

H

k

K

0

G

q

y

F

D

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

E

V

X

S

L

j

B

w

Y

M

o

v

Q

Z

u

d

O

U

>

A

7

3

c

N

J

2

+

T

C

p

I

g

W

/

m

R

P

z

9

5

8

r

n

f

H

k

K

0

G

q

y

F

D

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

E

V

X

S

L

j

B

w

Y

M

o

v

Q

Z

u

d

O

U

>

A

7

3

c

N

J

2

+

T

C

p

I

g

W

/

m

R

P

z

9

5

8

r

n

f

H

k

K

0

G

q

y

F

D

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

E

V

X

S

L

j

B

w

Y

M

o

v

Q

Z

u

d

O

U

>

A

7

3

c

N

J

2

+

T

C

p

I

g

W

/

m

R

P

z

9

5

8

r

n

f

H

k

K

0

G

q

y

F

D

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

O

U

c

/

J

Q

H

k

X

v

7

p

n

G

3

f

S

8

>

A

B

+

V

C

T

M

w

F

L

r

W

q

g

E

2

R

N

o

d

9

Y

I

z

u

P

D

j

5

m

K

y

Z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

O

U

c

/

J

Q

H

k

X

v

7

p

n

G

3

f

S

8

>

A

B

+

V

C

T

M

w

F

L

r

W

q

g

E

2

R

N

o

d

9

Y

I

z

u

P

D

j

5

m

K

y

Z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

O

U

c

/

J

Q

H

k

X

v

7

p

n

G

3

f

S

8

>

A

B

+

V

C

T

M

w

F

L

r

W

q

g

E

2

R

N

o

d

9

Y

I

z

u

P

D

j

5

m

K

y

Z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

O

U

c

/

J

Q

H

k

X

v

7

p

n

G

3

f

S

8

>

A

B

+

V

C

T

M

w

F

L

r

W

q

g

E

2

R

N

o

d

9

Y

I

z

u

P

D

j

5

m

K

y

Z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

m

w

9

2

B

V

8

I

q

z

A

C

g

k

K

>

7

3

c

N

S

E

J

U

r

/

L

p

5

Q

Y

F

o

+

u

D

H

G

Z

j

y

f

d

v

O

X

0

R

M

W

P

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

m

w

9

2

B

V

8

I

q

z

A

C

g

k

K

>

7

3

c

N

S

E

J

U

r

/

L

p

5

Q

Y

F

o

+

u

D

H

G

Z

j

y

f

d

v

O

X

0

R

M

W

P

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

m

w

9

2

B

V

8

I

q

z

A

C

g

k

K

>

7

3

c

N

S

E

J

U

r

/

L

p

5

Q

Y

F

o

+

u

D

H

G

Z

j

y

f

d

v

O

X

0

R

M

W

P

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

m

w

9

2

B

V

8

I

q

z

A

C

g

k

K

>

7

3

c

N

S

E

J

U

r

/

L

p

5

Q

Y

F

o

+

u

D

H

G

Z

j

y

f

d

v

O

X

0

R

M

W

P

n

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

o

H

f

U

T

y

V

7

d

5

O

n

c

X

Q

Z

>

A

B

N

S

8

E

J

r

q

/

9

L

C

p

I

Y

F

2

+

z

u

D

m

G

j

w

W

P

M

0

v

g

K

R

k

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

o

H

f

U

T

y

V

7

d

5

O

n

c

X

Q

Z

>

A

B

N

S

8

E

J

r

q

/

9

L

C

p

I

Y

F

2

+

z

u

D

m

G

j

w

W

P

M

0

v

g

K

R

k

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

o

H

f

U

T

y

V

7

d

5

O

n

c

X

Q

Z

>

A

B

N

S

8

E

J

r

q

/

9

L

C

p

I

Y

F

2

+

z

u

D

m

G

j

w

W

P

M

0

v

g

K

R

k

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

o

H

f

U

T

y

V

7

d

5

O

n

c

X

Q

Z

>

A

B

N

S

8

E

J

r

q

/

9

L

C

p

I

Y

F

2

+

z

u

D

m

G

j

w

W

P

M

0

v

g

K

R

k

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

E

I

B

Y

y

n

F

V

0

R

D

z

K

c

j

g

p

T

/

+

J

>

A

7

3

N

S

U

r

q

9

L

C

5

Q

2

o

u

H

m

G

Z

w

f

d

v

O

X

k

M

W

P

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

E

I

B

Y

y

n

F

V

0

R

D

z

K

c

j

g

p

T

/

+

J

>

A

7

3

N

S

U

r

q

9

L

C

5

Q

2

o

u

H

m

G

Z

w

f

d

v

O

X

k

M

W

P

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

E

I

B

Y

y

n

F

V

0

R

D

z

K

c

j

g

p

T

/

+

J

>

A

7

3

N

S

U

r

q

9

L

C

5

Q

2

o

u

H

m

G

Z

w

f

d

v

O

X

k

M

W

P

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

E

I

B

Y

y

n

F

V

0

R

D

z

K

c

j

g

p

T

/

+

J

>

A

7

3

N

S

U

r

q

9

L

C

5

Q

2

o

u

H

m

G

Z

w

f

d

v

O

X

k

M

W

P

φc (N (e)↪ N (e0 ))

is to a given neighbor of e. In another word, a neighbor of e is not required to be similar to all the neighbors of e′ , thus other similarities except the maximal one are discarded. Next, the maximal value smax i is transformed into a row-based feature vector Kr (Si), with each of the l-th element Kl(smax i ) being converted by the l-th RBF kernel with mean µl and variance σl. Then Kr (Si) of all the rows are averaged into a row-based similarity embedding ϕr (N (e), N (e′ )):

where n is the maximal number of neighbors over all the entities. S will be padded zero if the real number |N(e)| is less than n. The kernels are used to convert the one-dimension similarity into L-dimension similarities to enhance the discrimination ability, where the kernel with µ = 1 and σ → 0 only considers the exact matches between neighbors, and others capture the semantic matches between neighbors.

The above process aggregates the features along the rows, which reflects how similar each neighbor ei ∈ N(e) is to the neighbors N(e′ ) of e′ . Similarly, we also aggregate the features along the columns to capture how similar each neighbor e′j ∈ N(e′ ) is to the neighbors N(e) of e. Finally, the rowaggregation vector ϕr and the column-aggregation vector ϕc are concatenated as the final similarity embedding:

where ⊕ indicates the concatenation operation. Figure 3 illustrates the neighbor-view interactions.

Neighboring Relation Mask Matrix

A pair of neighbors are more convincing for supporting the alignment of two entities, if the corresponding relations are

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

P

8

j

E

f

T

d

S

w

G

u

3

U

y

Y

>

A

C

B

c

V

D

L

N

F

2

v

q

k

p

0

W

K

o

H

M

Q

Z

m

5

r

/

g

z

9

O

7

R

n

I

X

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

P

8

j

E

f

T

d

S

w

G

u

3

U

y

Y

>

A

C

B

c

V

D

L

N

F

2

v

q

k

p

0

W

K

o

H

M

Q

Z

m

5

r

/

g

z

9

O

7

R

n

I

X

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

P

8

j

E

f

T

d

S

w

G

u

3

U

y

Y

>

A

C

B

c

V

D

L

N

F

2

v

q

k

p

0

W

K

o

H

M

Q

Z

m

5

r

/

g

z

9

O

7

R

n

I

X

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

P

8

j

E

f

T

d

S

w

G

u

3

U

y

Y

>

A

C

B

c

V

D

L

N

F

2

v

q

k

p

0

W

K

o

H

M

Q

Z

m

5

r

/

g

z

9

O

7

R

n

I

X

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

q

d

F

L

X

7

N

W

9

I

T

B

Q

A

y

Y

k

>

C

c

V

D

S

J

3

U

2

v

E

o

u

H

P

G

O

f

0

M

z

r

+

/

n

j

p

g

m

K

R

w

5

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

q

d

F

L

X

7

N

W

9

I

T

B

Q

A

y

Y

k

>

C

c

V

D

S

J

3

U

2

v

E

o

u

H

P

G

O

f

0

M

z

r

+

/

n

j

p

g

m

K

R

w

5

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

q

d

F

L

X

7

N

W

9

I

T

B

Q

A

y

Y

k

>

C

c

V

D

S

J

3

U

2

v

E

o

u

H

P

G

O

f

0

M

z

r

+

/

n

j

p

g

m

K

R

w

5

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

q

d

F

L

X

7

N

W

9

I

T

B

Q

A

y

Y

k

>

C

c

V

D

S

J

3

U

2

v

E

o

u

H

P

G

O

f

0

M

z

r

+

/

n

j

p

g

m

K

R

w

5

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

j

+

v

T

K

W

w

E

O

L

U

P

B

c

M

k

r

Z

8

>

A

C

X

V

D

S

N

F

J

3

2

q

d

o

u

H

Q

G

Y

f

0

z

y

I

/

n

p

g

m

R

7

5

9

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

j

+

v

T

K

W

w

E

O

L

U

P

B

c

M

k

r

Z

8

>

A

C

X

V

D

S

N

F

J

3

2

q

d

o

u

H

Q

G

Y

f

0

z

y

I

/

n

p

g

m

R

7

5

9

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

j

+

v

T

K

W

w

E

O

L

U

P

B

c

M

k

r

Z

8

>

A

C

X

V

D

S

N

F

J

3

2

q

d

o

u

H

Q

G

Y

f

0

z

y

I

/

n

p

g

m

R

7

5

9

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

j

+

v

T

K

W

w

E

O

L

U

P

B

c

M

k

r

Z

8

>

A

C

X

V

D

S

N

F

J

3

2

q

d

o

u

H

Q

G

Y

f

0

z

y

I

/

n

p

g

m

R

7

5

9

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

c

I

B

X

U

q

H

K

0

7

T

5

D

3

z

M

W

J

>

A

C

V

S

g

N

F

2

r

f

Y

m

d

G

Z

L

O

u

Q

w

k

P

+

9

v

8

o

E

y

j

n

R

p

/

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

c

I

B

X

U

q

H

K

0

7

T

5

D

3

z

M

W

J

>

A

C

V

S

g

N

F

2

r

f

Y

m

d

G

Z

L

O

u

Q

w

k

P

+

9

v

8

o

E

y

j

n

R

p

/

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

c

I

B

X

U

q

H

K

0

7

T

5

D

3

z

M

W

J

>

A

C

V

S

g

N

F

2

r

f

Y

m

d

G

Z

L

O

u

Q

w

k

P

+

9

v

8

o

E

y

j

n

R

p

/

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

c

I

B

X

U

q

H

K

0

7

T

5

D

3

z

M

W

J

>

A

C

V

S

g

N

F

2

r

f

Y

m

d

G

Z

L

O

u

Q

w

k

P

+

9

v

8

o

E

y

j

n

R

p

/

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

O

U

c

/

J

Q

H

k

X

v

7

p

n

G

3

f

S

8

>

A

B

+

V

C

T

M

w

F

L

r

W

q

g

E

2

R

N

o

d

9

Y

I

z

u

P

D

j

5

m

K

y

Z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

O

U

c

/

J

Q

H

k

X

v

7

p

n

G

3

f

S

8

>

A

B

+

V

C

T

M

w

F

L

r

W

q

g

E

2

R

N

o

d

9

Y

I

z

u

P

D

j

5

m

K

y

Z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

O

U

c

/

J

Q

H

k

X

v

7

p

n

G

3

f

S

8

>

A

B

+

V

C

T

M

w

F

L

r

W

q

g

E

2

R

N

o

d

9

Y

I

z

u

P

D

j

5

m

K

y

Z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

O

U

c

/

J

Q

H

k

X

v

7

p

n

G

3

f

S

8

>

A

B

+

V

C

T

M

w

F

L

r

W

q

g

E

2

R

N

o

d

9

Y

I

z

u

P

D

j

5

m

K

y

Z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

y

w

c

r

L

n

v

D

H

7

8

C

2

3

A

/

j

O

+

M

>

B

V

S

g

N

E

f

U

Y

9

P

X

G

J

m

z

Z

R

k

q

I

T

o

K

u

W

0

p

F

Q

5

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

y

w

c

r

L

n

v

D

H

7

8

C

2

3

A

/

j

O

+

M

>

B

V

S

g

N

E

f

U

Y

9

P

X

G

J

m

z

Z

R

k

q

I

T

o

K

u

W

0

p

F

Q

5

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

y

w

c

r

L

n

v

D

H

7

8

C

2

3

A

/

j

O

+

M

>

B

V

S

g

N

E

f

U

Y

9

P

X

G

J

m

z

Z

R

k

q

I

T

o

K

u

W

0

p

F

Q

5

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

y

w

c

r

L

n

v

D

H

7

8

C

2

3

A

/

j

O

+

M

>

B

V

S

g

N

E

f

U

Y

9

P

X

G

J

m

z

Z

R

k

q

I

T

o

K

u

W

0

p

F

Q

5

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

z

F

0

Y

2

+

k

5

u

r

D

L

G

T

O

f

o

P

W

>

A

C

E

X

c

V

S

N

J

3

U

v

q

g

p

H

w

m

7

Q

n

/

B

j

8

M

Z

R

I

K

y

9

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

z

F

0

Y

2

+

k

5

u

r

D

L

G

T

O

f

o

P

W

>

A

C

E

X

c

V

S

N

J

3

U

v

q

g

p

H

w

m

7

Q

n

/

B

j

8

M

Z

R

I

K

y

9

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

z

F

0

Y

2

+

k

5

u

r

D

L

G

T

O

f

o

P

W

>

A

C

E

X

c

V

S

N

J

3

U

v

q

g

p

H

w

m

7

Q

n

/

B

j

8

M

Z

R

I

K

y

9

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

z

F

0

Y

2

+

k

5

u

r

D

L

G

T

O

f

o

P

W

>

A

C

E

X

c

V

S

N

J

3

U

v

q

g

p

H

w

m

7

Q

n

/

B

j

8

M

Z

R

I

K

y

9

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

2

Y

5

n

L

V

A

o

+

F

C

v

P

c

I

y

9

w

>

B

7

3

D

S

g

N

E

O

r

f

U

X

j

G

M

m

z

d

R

k

p

/

H

T

K

q

u

W

0

Z

Q

J

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

2

Y

5

n

L

V

A

o

+

F

C

v

P

c

I

y

9

w

>

B

7

3

D

S

g

N

E

O

r

f

U

X

j

G

M

m

z

d

R

k

p

/

H

T

K

q

u

W

0

Z

Q

J

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

2

Y

5

n

L

V

A

o

+

F

C

v

P

c

I

y

9

w

>

B

7

3

D

S

g

N

E

O

r

f

U

X

j

G

M

m

z

d

R

k

p

/

H

T

K

q

u

W

0

Z

Q

J

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

2

Y

5

n

L

V

A

o

+

F

C

v

P

c

I

y

9

w

>

B

7

3

D

S

g

N

E

O

r

f

U

X

j

G

M

m

z

d

R

k

p

/

H

T

K

q

u

W

0

Z

Q

J

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

O

r

G

n

v

z

K

H

2

w

V

5

I

Y

A

E

c

M

>

B

7

3

D

L

S

g

N

y

f

U

9

P

X

j

m

d

R

C

k

p

/

T

o

q

u

W

0

Z

F

Q

+

J

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

O

r

G

n

v

z

K

H

2

w

V

5

I

Y

A

E

c

M

>

B

7

3

D

L

S

g

N

y

f

U

9

P

X

j

m

d

R

C

k

p

/

T

o

q

u

W

0

Z

F

Q

+

J

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

O

r

G

n

v

z

K

H

2

w

V

5

I

Y

A

E

c

M

>

B

7

3

D

L

S

g

N

y

f

U

9

P

X

j

m

d

R

C

k

p

/

T

o

q

u

W

0

Z

F

Q

+

J

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

O

r

G

n

v

z

K

H

2

w

V

5

I

Y

A

E

c

M

>

B

7

3

D

L

S

g

N

y

f

U

9

P

X

j

m

d

R

C

k

p

/

T

o

q

u

W

0

Z

F

Q

+

J

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

7

o

0

Q

d

Y

m

O

8

B

G

5

p

>

A

3

c

V

N

S

w

M

E

J

2

X

+

T

C

I

g

L

j

W

/

Z

v

R

P

z

9

u

f

F

D

q

H

y

r

k

K

n

U

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

7

o

0

Q

d

Y

m

O

8

B

G

5

p

>

A

3

c

V

N

S

w

M

E

J

2

X

+

T

C

I

g

L

j

W

/

Z

v

R

P

z

9

u

f

F

D

q

H

y

r

k

K

n

U

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

7

o

0

Q

d

Y

m

O

8

B

G

5

p

>

A

3

c

V

N

S

w

M

E

J

2

X

+

T

C

I

g

L

j

W

/

Z

v

R

P

z

9

u

f

F

D

q

H

y

r

k

K

n

U

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

7

o

0

Q

d

Y

m

O

8

B

G

5

p

>

A

3

c

V

N

S

w

M

E

J

2

X

+

T

C

I

g

L

j

W

/

Z

v

R

P

z

9

u

f

F

D

q

H

y

r

k

K

n

U

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

S

X

V

/

Y

n

c

f

d

B

J

k

O

N

w

L

u

>

A

7

3

D

g

E

y

r

U

9

P

j

G

M

I

m

2

z

R

C

p

H

T

o

K

q

v

W

0

Z

F

Q

8

+

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

S

X

V

/

Y

n

c

f

d

B

J

k

O

N

w

L

u

>

A

7

3

D

g

E

y

r

U

9

P

j

G

M

I

m

2

z

R

C

p

H

T

o

K

q

v

W

0

Z

F

Q

8

+

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

S

X

V

/

Y

n

c

f

d

B

J

k

O

N

w

L

u

>

A

7

3

D

g

E

y

r

U

9

P

j

G

M

I

m

2

z

R

C

p

H

T

o

K

q

v

W

0

Z

F

Q

8

+

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

S

X

V

/

Y

n

c

f

d

B

J

k

O

N

w

L

u

>

A

7

3

D

g

E

y

r

U

9

P

j

G

M

I

m

2

z

R

C

p

H

T

o

K

q

v

W

0

Z

F

Q

8

+

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

N

j

w

y

m

Q

O

U

5

D

C

Z

K

p

S

E

7

>

A

B

3

c

V

L

g

r

f

Y

9

P

X

G

M

I

2

z

d

R

k

/

T

o

q

v

u

W

0

F

n

8

J

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

N

j

w

y

m

Q

O

U

5

D

C

Z

K

p

S

E

7

>

A

B

3

c

V

L

g

r

f

Y

9

P

X

G

M

I

2

z

d

R

k

/

T

o

q

v

u

W

0

F

n

8

J

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

N

j

w

y

m

Q

O

U

5

D

C

Z

K

p

S

E

7

>

A

B

3

c

V

L

g

r

f

Y

9

P

X

G

M

I

2

z

d

R

k

/

T

o

q

v

u

W

0

F

n

8

J

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

N

j

w

y

m

Q

O

U

5

D

C

Z

K

p

S

E

7

>

A

B

3

c

V

L

g

r

f

Y

9

P

X

G

M

I

2

z

d

R

k

/

T

o

q

v

u

W

0

F

n

8

J

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

E

7

j

g

/

k

H

8

I

N

d

D

n

3

o

X

v

Z

M

>

A

B

c

V

S

w

J

2

+

T

C

p

L

W

Q

m

R

P

z

9

u

5

Y

r

O

f

G

q

K

F

U

y

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

E

7

j

g

/

k

H

8

I

N

d

D

n

3

o

X

v

Z

M

>

A

B

c

V

S

w

J

2

+

T

C

p

L

W

Q

m

R

P

z

9

u

5

Y

r

O

f

G

q

K

F

U

y

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

E

7

j

g

/

k

H

8

I

N

d

D

n

3

o

X

v

Z

M

>

A

B

c

V

S

w

J

2

+

T

C

p

L

W

Q

m

R

P

z

9

u

5

Y

r

O

f

G

q

K

F

U

y

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

E

7

j

g

/

k

H

8

I

N

d

D

n

3

o

X

v

Z

M

>

A

B

c

V

S

w

J

2

+

T

C

p

L

W

Q

m

R

P

z

9

u

5

Y

r

O

f

G

q

K

F

U

y

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

f

w

9

N

T

Z

d

W

A

B

U

k

+

n

Q

o

V

I

>

7

3

c

D

L

S

g

E

O

y

r

Y

P

X

j

G

M

m

2

z

R

C

p

/

H

K

q

v

u

0

F

8

J

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

f

w

9

N

T

Z

d

W

A

B

U

k

+

n

Q

o

V

I

>

7

3

c

D

L

S

g

E

O

y

r

Y

P

X

j

G

M

m

2

z

R

C

p

/

H

K

q

v

u

0

F

8

J

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

f

w

9

N

T

Z

d

W

A

B

U

k

+

n

Q

o

V

I

>

7

3

c

D

L

S

g

E

O

y

r

Y

P

X

j

G

M

m

2

z

R

C

p

/

H

K

q

v

u

0

F

8

J

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

f

w

9

N

T

Z

d

W

A

B

U

k

+

n

Q

o

V

I

>

7

3

c

D

L

S

g

E

O

y

r

Y

P

X

j

G

M

m

2

z

R

C

p

/

H

K

q

v

u

0

F

8

J

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

2

O

S

E

+

/

Q

D

Z

k

B

u

C

L

y

0

I

w

>

A

7

3

c

V

g

N

r

f

U

Y

9

P

X

j

G

M

m

z

d

R

p

H

o

K

q

v

W

F

n

J

5

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

2

O

S

E

+

/

Q

D

Z

k

B

u

C

L

y

0

I

w

>

A

7

3

c

V

g

N

r

f

U

Y

9

P

X

j

G

M

m

z

d

R

p

H

o

K

q

v

W

F

n

J

5

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

2

O

S

E

+

/

Q

D

Z

k

B

u

C

L

y

0

I

w

>

A

7

3

c

V

g

N

r

f

U

Y

9

P

X

j

G

M

m

z

d

R

p

H

o

K

q

v

W

F

n

J

5

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

2

O

S

E

+

/

Q

D

Z

k

B

u

C

L

y

0

I

w

>

A

7

3

c

V

g

N

r

f

U

Y

9

P

X

j

G

M

m

z

d

R

p

H

o

K

q

v

W

F

n

J

5

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

E

k

M

f

U

2

q

7

P

3

r

Q

5

C

W

>

A

B

c

V

D

L

S

g

N

O

y

Y

9

X

j

G

I

m

z

R

p

/

w

H

T

o

K

v

u

0

Z

F

n

J

+

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

E

k

M

f

U

2

q

7

P

3

r

Q

5

C

W

>

A

B

c

V

D

L

S

g

N

O

y

Y

9

X

j

G

I

m

z

R

p

/

w

H

T

o

K

v

u

0

Z

F

n

J

+

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

E

k

M

f

U

2

q

7

P

3

r

Q

5

C

W

>

A

B

c

V

D

L

S

g

N

O

y

Y

9

X

j

G

I

m

z

R

p

/

w

H

T

o

K

v

u

0

Z

F

n

J

+

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

E

k

M

f

U

2

q

7

P

3

r

Q

5

C

W

>

A

B

c

V

D

L

S

g

N

O

y

Y

9

X

j

G

I

m

z

R

p

/

w

H

T

o

K

v

u

0

Z

F

n

J

+

8

also similar. In another word, for a triplet (e, ri, ei) of e and a triplet (e′ , r′j , e′j ) of e′ , if ei is very similar to e′j and ri is also similar to r′j , the two entities e and e′ will be more likely to be aligned. According to this assumption, we not only compute the similarity matrix S between the neighboring entities but also compute the similarity matrix M between the corresponding neighboring relations. M is viewed as a mask matrix and is multiplied to S, i.e., Sij = Sij ⊗Mij, where ⊗ indicates elementwise product. To compute M, we need to embed each neighboring relation. One popular way is to represent a relation approximately by its associated head-tail entity pairs based on the assumption that two relations are more similar if they associate to more similar head-tail pairs [Wu et al., 2019a]. Specifically, we average C(e) of all the associated head entities and also average those of all the tail entities, and concatenate them to represent the relation.

Interactions between Multi-hop Neighbors

Intuitively, the one-hop neighbors take the most important role in aligning two entities. However, as claimed by [Sun et al., 2020], the multi-hop neighbors also impact the alignment results in some situations, as the direct neighbors of one entity in G will probably appear as the distant neighbors of the counterpart in G′ due to the heterogeneity of two KGs. Thus, we also consider the interactions between multihop neighbors. Specifically, given one-hop neighbors N 1 (e) and N 1 (e′ ) and multi-hop neighbors N m(e) and N m(e′ ), we build the interaction matrix between N 1 (e) and N 1 (e′ ), N 1 (e) and N m(e′ ), N m(e) and N 1 (e′ ), and N m(e) and N m(e′ ), use the same aggregation functions as Eq.(3) and Eq.(4) to extract the similarity vector respectively, and concatenate them as the final neighbor similarity embedding.

Attribute-view Interactions

We also build the interactions between the attributes A(e) and A(e′ ). Different from a unique name or description, attributes are a set of attribute-value pairs, which is similar to the neighboring relation-entity pairs. Thus similar to the neighbor-view interactions, we also compare each attribute pair rather than learning a global representation for e or e′ by aggregating all their attributes as existing works did [Sun et al., 2017; Zhang et al., 2019]. Specifically, in Figure 3, we change the similarity matrix between the neighboring entities into that between the embeddings {C(vi)}|A(e)| i=1 of e's values and the embeddings {C(v′i )}|A(e′ )| i=1 of e′ 's values, and change the mask matrix between the neighboring relations into that between the embeddings {C(ai)}|A(e)| i=1 of e's attributes and the embeddings {C(a′i )}|A(e′ )| i=1 of e′ 's attributes. The assumption is a pair of values is more convincing for supporting the alignment of two entities if the corresponding attributes are also similar. Then we use the same aggregation functions as Eq.(3) and Eq.(4) to produce an attribute similarity vector ϕ(A(e),A(e′ )). Note we ignore the attributes of neighbors, as on one hand, incorporating neighbors' attributes will result in nested interactions, which is inefficient. On the other hand, for aligning e and e′ , their own attributes are most important.

The Final Combination

Given the cosine similarity cos(C(e),C(e′ )) between the descriptions/names of two entities, the neighbor similarity vector ϕ(N(e),N(e′ )) and the attribute similarity vector ϕ(A(e),A(e′ )), we concatenate them together and apply a MLP layer to get the final similarity score between e and e′ :

Finally, we inject g(e, e′ ) into the same pairwise loss function as Eq.(2) to optimize the parameters of MLP. Note that the parameters of BERT have been fine-tuned by the basic BERT unit in Section 3.1 and are frozen at this stage.

3.3 Entity Alignment

Given an entity e from G, we first quickly filter top-κ candidates fromG′ and then accurately infer the counterpart from the candidates. Specifically, we apply the basic BERT unit to obtain an embedding for each entity by Eq.(1), compute the cosine similarity between the embedding of e from G and the embedding of each entity from G′ , and return the top-κ similar entities as candidates of e. Then for e and each candidate, we apply the BERT-based interaction model to infer a matching score between them and rank all the candidates for evaluation. The candidate selection process can significantly improve the alignment efficiency by the interaction model.

4 Experiments

4.1 Datasets and Settings

We evaluate our model on the widely used cross-lingual dataset DBP15K and the mono-lingual dataset DWY100K and use HitRatio@K (K=1,10) and MRR to evaluate (Cf. [Sun et al., 2018] for details). The dimension of the BERT CLS embedding is 768. We use a 300-dimension MLP in Eq.(1) and a 11 plus 1-dimension MLP in Eq.(5). The maximal number of neighbors and attributes are both set as 50. In Eq.(3), we use 20 semantic matching kernels, where µ is from 0.025 to 0.975 with interval 0.05 and all σ = 0.1, and use an exact matching kernel with µ = 1.0 and σ = 10-3 . The number of the returned candidates by the basic BERT unit, i.e., κ is set as 50, as we find that 99% ground truth can be included in the top-50 candidates. The margin m in Eq.(2) for fine-tuning BERT is set as 3, and for training the interaction model is set as 1. Codes and datasets are online now 3 .

4.2 Experimental Results Overall Performance on DBP15K

We compare all the state-of-the-art models with available results or codes. Some methods such as [Chen et al., 2018; Trsedya et al., 2019] are not compared due to the code implementation issue. In principle, we divide them into the category that only utilizes graph structures and the one that uses additional side information. We further divide the former category into three fine-grained types: variant TransE, variant TransE plus GCN, and variant TransE plus adversarial learning, and divide the later one into two fine-grained types: variant GCN and multi-view learning. Table 1 shows the overall performance on DBP15K. Generally, we conclude that including side information can obtain better performance than only considering graph structures.

3 https://github.com/kosugi11037/bert-int

Table 1: Overall performance of entity alignment on DBP15K.

	DBP15KZH-EN		DBP15KJA-EN DBP15KFR-EN
Model	HR1 HR10 MRR	HR1 HR10 MRR	HR1	HR10 MRR
Only use graph structures by variant TransE
MTransE [Chen et al., 2017]	0.308 0.614 0.364	0.279 0.575 0.349	0.244	0.556 0.335
IPTransE [Zhu et al., 2017]	0.406 0.735 0.516	0.367 0.693 0.474	0.333	0.685 0.451
BootEA [Sun et al., 2018]	0.629 0.848 0.703	0.622 0.854 0.701	0.653	0.874 0.731
RSNs [Guo et al., 2019]	0.508 0.745 0.591	0.507 0.737 0.590	0.516	0.768 0.605
TransEdge [Sun et al., 2019]	0.735 0.919 0.801	0.719 0.932 0.795	0.710	0.941 0.796
MRPEA [Shi and Xiao, 2019]	0.681 0.867 0.748	0.655 0.859 0.727	0.677	0.890 0.755
Only use graph structures by variant TransE plus GCN
MuGNN [Cao et al., 2019]	0.494 0.844 0.611	0.501 0.857 0.621	0.495 0.870	0.621
NAEA [Zhu et al., 2019]	0.650 0.867 0.720	0.641 0.873 0.718	0.673 0.894	0.752
KECG [Li et al., 2019]	0.478 0.835 0.598	0.490 0.844 0.610	0.486 0.851	0.610
AliNet [Sun et al., 2020]	0.539 0.826 0.628	0.549 0.831 0.645	0.552	0.852 0.657
Only use graph structures by variant TransE plus adversarial learning
AKE [Lin et al., 2019]	0.325 0.703 0.449	0.259 0.663 0.390	0.287 0.681	0.416
SEA [Pei et al., 2019]	0.424 0.796 0.548	0.385 0.783 0.518	0.400	0.797 0.533
Combine graph structures and side information by variant GCN
GCN-Align [Wang et al., 2018]	0.413 0.744 0.549	0.399 0.745 0.546	0.373 0.745	0.532
GM-Align [Xu et al., 2019]	0.679 0.785 -	0.740 0.872 -	0.894 0.952	-
RDGCN [Wu et al., 2019a]	0.708 0.846 0.746	0.767 0.895 0.812	0.886	0.957 0.911
HGCN [Wu et al., 2019b]	0.720 0.857 0.768	0.766 0.897 0.813	0.892 0.961	0.917
DGMC [Fey et al., 2020]	0.772 0.897 -	0.774 0.907 -	0.891	0.967 -
Combine graph structures and side information by multi-view learning
JAPE [Sun et al., 2017]	0.412 0.745 0.490	0.363 0.685 0.476	0.324	0.667 0.430
MultiKE [Zhang et al., 2019]	0.509 0.576 0.532	0.393 0.489 0.426	0.639	0.712 0.665
JarKA [Chen et al., 2020]	0.706 0.878 0.766	0.646 0.855 0.708	0.704	0.888 0.768
HMAN [Yang et al., 2019]	0.871 0.987 -	0.935 0.994 -	0.973	0.998 -
CEAFF [Zeng et al., 2020]	0.795 - -	0.860 - -	0.964	- -
BERT-INT	0.968 0.990 0.977	0.964 0.991 0.975	0.992	0.998 0.995

Only using Graph Structures. We analyze the differences of the methods only using the graph structures. Among variant TransE methods, BootEA, TransEdge and MRPEA perform the best, where BootEA and TransEdge both bootstrap the labeled alignments iteratively, which is the key technique to improve the alignment performance. Strictly, MRPEA is a little far away from TransE, as it changes the subtraction between entities and relations into multiplications, which is more friendly to the multi-mapping relations. Several works combine the advantages of TransE and GCN by convolving all the neighboring information for an entity as GCN does and meanwhile keeping the translation relationship among head, relation and tail as TransE does. Among them, NAEA also bootstraps the labeled alignments. For other methods without bootstrapping, we cannot observe significant improvement by incorporating GCN. This may due to the heterogeneity of KGs. Two neighboring entities in KGs are more likely to be translated to each other by their relation, rather than similar to each other as assumed by GCN. Besides, we notice that based on TransE, incorporating adversarial learning cannot obtain expected performance. This is also caused by the high heterogeneity of KGs, which results in the difficulty of transferring from one KG to another by a similar linear mapping function as transferring between multi-lingual word spaces [Conneau et al., 2018].

CombiningGraph Structures and Side Information. We analyze the differences of the methods that also incorporate the side information. Among the methods that model the graph structures and the side information in a unified GCN model, GM-Align, RDGCN, and HGCN distinguish the effects of different neighbors, thus they perform better than the original GCN-Align model. Among the methods that model the graph structures and the side information separately and then combine them by multi-view learning, CEAFF that emphasizes the effect of name and HMAN that uses BERT to embed descriptions, significantly outperform others. DGMC and CEAFF both solve the global alignment inconsistency and obtain excellent performance. CEAFF further outperforms DGMC by 2.3-8.6% in HR1, which indicates that separating the two views can reduce noises and is more reasonable than mixing the two views in a unified GCN network.

Only using Side Information. Our model BERT-INT only uses the side information. Despite the missing of graph structures, BERT-INT outperforms the best baselines by 9.7%, 2.9% and 1.9% in HR1 on ZH-EN, JA-EN, and FR-EN respectively, which indicates that the semantics of the side information is more powerful than the structural characteristics. Compared with all the other methods, we directly compare neighbors or attributes through the interaction mechanism, which can improve the matching accuracy between entities.

Overall Performance on DWY100K

Since CEAFF [Zeng et al., 2020] has already obtained 100% HR1 on the mono-lingual dataset DWY100K, we only compare our model with it. We use the name as the basic representation of an entity due to the missing descriptions in the dataset. On the two datasets of DWY100K, we obtain 99.2% and 99.9% HR1 respectively, which are comparable to CEAFF. However, we also observe that the names for most of the aligned entities are exactly the same, which demands more challenging mono-lingual datasets [Zeng et al., 2020].

Ablation Study

We perform ablation study from three aspects and show the results in Table 2. All the variants are based on BERT-INT.

Remove Components from BERT-INT. When removing the max-pooling operations from the interaction model 4 , the performance is reduced by maximal 0.6%, which indicates it is more effective to only capture the most possibly aligned counterpart rather than taking the similarities to all the neighbors/attributes into consideration. When removing the column-aggregation 5 , the performance is reduced by maximal 0.8%, which indicates that we need to consider the similarity of each neighbor to all the corresponding neighbors from both the directions. When removing the neighbors or the attributes, the performance is reduced by maximal 2.7% or 4.9% respectively, and significantly reduced by maximal 13.8%when removing both of them, i.e., only using the names/descriptions of entities. The results demonstrate modeling the interactions of neighbors and attributes is effective.

4 We remove both the row-based and column-based max-pooling.

Table 2: Ablation study on DBP15K.

	DBP15KZH-EN			DBP15KJA-EN			DBP15KFR-EN
Model	HR1	HR10	MRR	HR1	HR10	MRR	HR1	HR10	MRR
BERT-INT	0.968	0.990	0.977	0.964	0.991	0.975	0.992	0.998	0.995
Remove components
-max pooling -column aggregation	0.962	0.989 0.989	0.973 0.971 0.963	0.959	0.991	0.973	0.992	0.998	0.995
	0.960			0.959	0.990	0.971	0.991	0.998	0.994
-neighbors	0.947	0.987		0.937	0.986	0.956	0.988	0.998	0.992
-attributes	0.919	0.984	0.945	0.938	0.987	0.957	0.983	0.998	0.990
-neighbors & attributes	0.830	0.970	0.883	0.848	0.974	0.897	0.965	0.995	0.978
Change the interaction component to variant GCN
BERT-GCN	0.736	0.950	0.799	0.767	0.960	0.824	0.914	0.992	0.936
BERT-RDGCN	0.847	0.974	0.896	0.857	0.969	0.900	0.952	0.990	0.967
BERT-HMAN	0.911	0.993	0.943	0.937	0.994	0.960	0.982	0.999	0.989
Add components
+relation mask	0.966	0.989	0.975	0.962	0.990	0.973	0.992	0.998	0.995
+attribute mask	0.942	0.986	0.959	0.950	0.990	0.966	0.989	0.998	0.993
+2-hop neighbors	0.965	0.990	0.975	0.964	0.991	0.975	0.992	0.998	0.995

Change Interaction Model in BERT-INT to GCN. We initialize each node e by C(e) in Eq.(1), and update node embeddings by the original GCN or RDGCN [Wu et al., 2019a]. Compared with BERT-INT, HR1 is reduced by maximal 23.2%, and is even 1.3-9.4% worse than -neighbors & attributes, i.e., our model that only uses the names/descriptions of entities. The results indicate that when the side information is powerfully modeled by BERT, introducing them in GCNs may lose effect, and even introduce noises. For fair comparison, we change the BERT embeddings in HMAN by ours [Yang et al., 2019]. The result of BERT-HMAN also under-performs BERT-INT by maximal 5.7% HR1, which reveals the effectiveness of the interaction part in BERT-INT.

Add Components into BERT-INT. BERT-INT does not include the relation/attribute mask matrix and the multi-hop neighbors. We add each of them to BERT-INT to validate the effect. Unfortunately, none of them present an improvement on BERT-INT. We analyze the reasons as follows. Following [Wu et al., 2019a], we represent a relation by all its associated head-tail entities, which is too general to reflect the exact meaning of the relation. We also tried to represent a relation by its name but obtained the similar results, as names are too short to represent a relation completely. Attributes can be explained in the same way. To test the effect of multi-hop neighbors, we take 2-hop neighbors as an example. As some entities have a large number of 2-hop neighbors, to reduce the noises, for each pair of entities, we select top-k similar 2hop neighbor pairs based on their BERT embeddings 6 . As the BERT-based neighbor/attribute interactions are already powerful, it limits the effect of the 2-hop neighbors. Although AliNet [Sun et al., 2020] shows the effectiveness of the multihop neighbors, it only uses the structure embeddings, which are not as powerful as BERT embeddings, thus it needs more structural information from the multi-hop neighbors.

5 Removing row-aggregation obtains the similar results.

Inductive Learning

We train the basic BERT unit and BERT-INT on one dataset of DBP15K, directly transfer and evaluate them on the other two datasets, and show the results in Figure 4(a) and Figure 4(b). When training BERT-INT on ZH-EN/JA-EN and testing on JA-EN/ZH-EN, the results are even better than HMAN and CEAFF, which reveals ZH and JA are easier to be transferred between each other. No matter training on any dataset, the test performance on FR-EN is comparable to the normal one (i.e., train and test on FR-EN), because FR is similar to EN, making the alignment avoids the affect of different languages. We also observe that the basic BERT unit also has the inductive capacity. However, on the basis of the normal performance on JA-EN, the reduced HR1 by the transferred BERT-INT is 18.8% less than that by the the transferred basic BERT unit, which reveals the inductive capacity is not only caused by multi-lingual BERT, but also caused by the proposed interaction model.

5 Conclusions

This paper solves knowledge graph alignment by building the interactions between neighbors or attributes based on their BERT embeddings, which can obtain fine-grained matches of neighbors or attributes. The proposed model can achieve the best performance among all the state-of-the-art models and can enable inductive learning compared with other models.

Acknowledgments

This work is supported by National Key R&D Program of China (No.2018YFB1004401) and NSFC (No.61532021, 61772537, 61772536, 61702522)

6 We vary k from 10 to 200 and find the best value as 50.