MDNN: A Multimodal Deep Neural Network for Predicting Drug-Drug Interaction Events
Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 3536-3542.
https://doi.org/10.24963/ijcai.2021/487
Keywords
- Machine Learning Applications: Bio/Medicine
- Multidisciplinary Topics and Applications: Biology and Medicine
- Multidisciplinary Topics and Applications: AI for Life Science
Abstract
The interaction of multiple drugs could lead to serious events, which causes injuries and huge medical costs. Accurate prediction of drug-drug interaction (DDI) events can help clinicians make effective decisions and establish appropriate therapy programs. Recently, many AI-based techniques have been proposed for predicting DDI associated events. However, most existing methods pay less attention to the potential correlations between DDI events and other multimodal data such as targets and enzymes. To address this problem, we propose a Multimodal Deep Neural Network (MDNN) for DDI events prediction. In MDNN, we design a two-pathway framework including drug knowledge graph (DKG) based pathway and heterogeneous feature (HF) based pathway to obtain drug multimodal representations. Finally, a multimodal fusion neural layer is designed to explore the complementary among the drug multimodal representations. We conduct extensive experiments on real-world dataset. The results show that MDNN can accurately predict DDI events and outperform the state-of-the-art models.
- [Abdelaziz et al., 2017] Ibrahim Abdelaziz, Achille Fok- oue, Oktie Hassanzadeh, Ping Zhang, and Mohammad Sadoghi. Large-scale structural and textual similarity- based mining of knowledge graph to predict drug-drug in- teractions. Journal of Web Semantics, 44:104-117, 2017.
- [Chu et al., 2019] Xu Chu, Yang Lin, Yasha Wang, Leye Wang, Jiangtao Wang, and Jingyue Gao. Mlrda: A multi- task semi-supervised learning framework for drug-drug in- teraction prediction. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 4518- 4524, 2019.
- [Cui et al., 2020] Limeng Cui, Haeseung Seo, Maryam Tabar, Fenglong Ma, Suhang Wang, and Dongwon Lee. Deterrent: Knowledge guided graph attention network for detecting healthcare misinformation. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 492-502, 2020.
- [Deng et al., 2020] Yifan Deng, Xinran Xu, Yang Qiu, Jingbo Xia, Wen Zhang, and Shichao Liu. A multimodal deep learning framework for predicting drug-drug interac- tion events. Bioinformatics, 36:4316-4322, 2020.
- [Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing sys- tems, pages 1024-1034, 2017.
- [Hao et al., 2020] Zhongkai Hao, Chengqiang Lu, Zhenya Huang, Hao Wang, Zheyuan Hu, Qi Liu, Enhong Chen, and Cheekong Lee. Asgn: An active semi-supervised graph neural network for molecular property prediction. In Proceedings of the ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, pages 731-752, 2020.
- [Huang et al., 2020] Kexin Huang, Cao Xiao, Trong Hoang, Lucas Glass, and Jimeng Sun. Caster: Predicting drug in- teractions with chemical substructure representation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 702-709, 2020.
- [Jia et al., 2020] Ziyu Jia, Youfang Lin, Jing Wang, Rong- hao Zhou, Xiaojun Ning, Yuanlai He, and Yaoshuai Zhao. Graphsleepnet: Adaptive spatial-temporal graph convolu- tional networks for sleep stage classification. In Proceed- ings of the International Joint Conference on Artificial In- telligence, pages 1324-1330, 2020. IJCAI
- [Jin et al., 2017] Bo Jin, Haoyu Yang, Cao Xiao, Ping Zhang, Xiaopeng Wei, and Fei Wang. Multitask dyadic prediction and its application in prediction of adverse drug-drug interaction. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 31, pages 1367- 1373, 2017.
- [Lin et al., 2020] Xuan Lin, Zhe Quan, Zhi-Jie Wang, Tengfei Ma, and Xiangxiang Zeng. Kgnn: Knowledge graph neural network for drug-drug interaction prediction. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 2739-2745, 2020. IJCAI
- [Ma et al., 2018] Tengfei Ma, Cao Xiao, Jiayu Zhou, and Fei Wang. Drug similarity integration through attentive multi- view graph auto-encoders. In Proceedings of the Inter- national Joint Conference on Artificial Intelligence, pages 3477-3483, 2018.
- [Ryu et al., 2018] Jae Yong Ryu, Hyun Uk Kim, and Sang Yup Lee. Deep learning improves prediction of drug- drug and drug-food interactions. Proceedings of the Na- tional Academy of Sciences, 115(18):E4304-E4311, 2018.
- [Song et al., 2020] Ying Song, Shuangjia Zheng, Zhang- ming Niu, Zhang-Hua Fu, Yutong Lu, and Yuedong Yang. Communicative representation learning on at- tributed molecular graphs. In Proceedings of the Inter- national Joint Conference on Artificial Intelligence, pages 2831-2838, 2020. IJCAI
- [Vilar et al., 2014] Santiago Vilar, Eugenio Uriarte, Lourdes Santana, Tal Lorberbaum, George Hripcsak, Carol Fried- man, and Nicholas P Tatonetti. Similarity-based modeling in large-scale prediction of drug-drug interactions. Nature protocols, 9(9):2147-2163, 2014.
- [Wang et al., 2019a] Hongwei Wang, Miao Zhao, Xing Xie, Wenjie Li, and Minyi Guo. Knowledge graph convolu- tional networks for recommender systems. In Proceed- ings of the World Wide Web Conference, pages 3307-3313, 2019.
- [Wang et al., 2019b] Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. Kgat: Knowledge graph attention network for recommendation. In Proceedings of the ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 950-958, 2019.
- [Yu et al., 2021] Yue Yu, Kexin Huang, Chao Zhang, Lu- cas M Glass, Jimeng Sun, and Cao Xiao. Sumgnn: Multi- typed drug interaction prediction via efficient knowledge graph summarization. Bioinformatics, pages 1-8, 2021.
- [Yue et al., 2020] Xiang Yue, Zhen Wang, Jingong Huang, Srinivasan Parthasarathy, Soheil Moosavinasab, Yungui Huang, Simon M Lin, Wen Zhang, Ping Zhang, and Huan Sun. Graph embedding on biomedical networks: methods, applications and evaluations. Bioinformatics, 36(4):1241- 1251, 2020.
- [Zhang et al., 2015] Ping Zhang, Fei Wang, Jianying Hu, and Robert Sorrentino. Label propagation prediction of drug-drug interactions based on clinical side effects. Sci- entific reports, 5(1):1-10, 2015.
- [Zhao et al., 2020] Yang Zhao, Jiajun Zhang, Yu Zhou, and Chengqing Zong. Knowledge graphs enhanced neural ma- chine translation. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 4039-4045, 2020. IJCAI
- [Zitnik et al., 2018] Marinka Zitnik, Monica Agrawal, and Jure Leskovec. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34(13):i457-i466, 2018.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Introduction
With the rapid growth of the number of drug types, it is essential to manage drug safety when multiple drugs are adopted in the treatment of a disease. Drug-Drug Interactions (DDI) often occur in cases of simultaneous administration of multiple drugs, which may result in adverse drug reactions that cause injuries and huge medical costs [Vilar et al., 2014]. However, DDI can lead to different biological consequences and events. For example, drug Itraconazole and drug Abemaciclib interaction together cause an event that the risk to increase due to the severity of the adverse effects, as shown in Figure 1. Therefore, accurate prediction of DDI events becomes a clinically important task which could help clinicians make effective decisions and establish appropriate therapy programs.
∗Corresponding Author
The correct use of multiple drugs can minimize the medical risks while maximizing the synergy benefits of drugs.
There have been a number of AI-based models proposed for DDI events prediction, including analyzing chemical structure similarity using graph neural networks [Huang et al., 2020], implementing multi-task learning on DDI type prediction [Jin et al., 2017; Zitnik et al., 2018; Ryu et al., 2018], modeling semi-supervised learning to mine useful information for DDI prediction in both labeled and unlabeled drug data [Chu et al., 2019], and exploiting knowledge graph summarization for multi-typed DDI pharmacological effect prediction [Yu et al., 2021]. There have been also some efforts on predicting DDI using multiple data sources, such as the similarity features to obtain drug features for DDI events prediction task [Ma et al., 2018; Zhang et al., 2015; Deng et al., 2020]. However, most existing methods pay less attention to the potential correlations between DDI events and other multimodal data such as targets and enzymes. Moreover, cross-modality complementarity of multimodal data has not been taken into consideration.
To tackle the above limitations, this work aims to effectively assist the joint representation learning of multimodal data related to DDI events. We propose a Multimodal Deep Neural Network (MDNN) framework for DDI events prediction. In MDNN, we design a two-pathway framework including drug knowledge graph (DKG) based pathway and heterogeneous feature (HF) based pathway to obtain drug multimodal representations. Then, inspired by graph neural networks that try to learn from structure information [Hamilton et al., 2017; Wang et al., 2019a; Cui et al., 2020], we propose the GNN layer to learn drug representations by extracting both structural information and semantic relations from the DKG. Finally, a multimodal fusion neural layer is designed to predict DDI events by exploring the complementary between the drug multimodal representations. Our contributions are summarized as follows:
- We propose a new multimodal deep neural network with a two-pathway framework including the drug knowledge graph pathway and the heterogeneous feature pathway. MDNN can predict DDI events by exploiting the associations between DDI events and multimodal representations.
- The MDNN framework mainly has the following merits: (a) MDNN learns the representations from multimodal data and mines the inter-modality similarities from multiple sources. (b) MDNN exploits the topological structure information and semantic relations with drug knowledge graph.
- We conduct extensive experiments on a real-world dataset to demonstrate the effectiveness of our model compared with classic and the state-of-the-art methods.
2 RelatedWork
DDI events prediction is a fundamental task with applications in many areas such as clinical and pharmaceutical decisions. The research works which aim to improve DDI prediction can be summarized in two directions: integrating multiple drug features and applying deep learning techniques.
Many efforts have been taken on calculating the similarities by integrating multiple data sources and predicting DDI based on the fused similarity. For example, the works [Vilar et al., 2014; Abdelaziz et al., 2017] integrate multiple drug features to calculate the similarities among drugs , and then predict DDI accurately based on the fused similarity. [Zhang et al., 2015] proposes an integrative framework to fuse the similarities of drug features with proper weights and predict DDI. [Ma et al., 2018] proposes to learn accurate and interpretable similarity measurement from multiple types of drug features for DDI prediction. In addition, [Deng et al., 2020] proposes a framework DDIMDL that combines diverse drug features to build a model for predicting DDI events. However, they are limited in obtaining the rich features of drugs in structural information and semantic relations.
Recently, there has been growing interests in applying AI techniques for DDI prediction such as deep learning and graph neural networks. Different from drug similarity obtained from multiple sources, a deep learning framework named DeepDDI [Ryu et al., 2018] is proposed to use molecular structures of drugs as inputs for predicting DDI types. The work [Jin et al., 2017] proposes a new multitask dyadic prediction model to predict adverse drug-drug interactions.
MLRDA [Chu et al., 2019] develops a multi-task semisupervised learning framework which effectively exploits information that is beneficial for DDI prediction in unlabeled drug data.
Inspired by the success of applying graph neural networks (GNN) in a wide variety of tasks [Jia et al., 2020; Song et al., 2020; Hao et al., 2020], researchers also tried to utilize GNN to improve the performance of DDI events prediction. For example, Decagon [Zitnik et al., 2018] applies a relational GNN for predicting side effects of drug pairs. [Yue et al., 2020] integrates graph embedding methods for DDI prediction task. In addition, CASTER [Huang et al., 2020] develops an end-to-end dictionary learning framework for predicting DDI with chemical structures of drugs. KGNN [Lin et al., 2020] designs an effective framework for DDI prediction which can capture drug and its potential neighborhoods in the knowledge graph. Although these methods have achieved relatively good performance, they do not consider the drug multimodal data coherence and complementarity together. In addition, knowledge graph can provide a large amount of structured information among multiple entities and semantic relations associated with entities. Knowledge graphs are a powerful tool [Zhao et al., 2020; Wang et al., 2019b], and some biomedical knowledge bases have been published in this form. These knowledge graphbased methods also have been used in structured scenarios of DDI prediction [Lin et al., 2020]. However, most of these methods ignore the multimodal data. Moreover, only a few methods take different drug features as independent data and do not take cross-modality complementarity into consideration. Compared with these methods, our model uses a newly designed graph neural network to capture both the topological information and semantic relations, and explores the crossmodality complementarity of multimodal data, which differentiates it from the existing methods.
3 ProblemFormulation
In this section, we formulate the problem of DDI events prediction that we will tackle. We first present several basic definitions which will be used in the problem formulation.
DDI Matrix. Formally, we denote DDI events Y ∈ (0, yij)Nd×Nd as the label matrix for this prediction task, where Nd denotes the number of drugs in the DDI events matrix. yij ∈ L is a label, where L = {y1, y2, · · · , yNl } denotes the label set and Nl denotes the types number of events. For each DDI event, yij ∈ L means that the interaction event yij exists between drug di and drug dj, and yij = 0 means that there is no interaction event existing between drug di and drug dj .
Drug Knowledge Graph (DKG). We consider a special type of knowledge graph for DDI events prediction named drug knowledge graph (DKG), denoted by G = (D, R, T ):
where D and T describe a subset of drug entities and a subset of tail entities (drug related nodes, e.g. targets) respectively, and R denotes the set of relations between drugs and tail entities.
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
N
o
O
d
2
M
p
v
H
9
5
W
+
D
Z
V
k
>
A
B
n
c
L
S
g
E
y
r
f
U
Y
P
K
Q
I
X
G
j
7
m
F
T
/
w
q
0
8
3
u
C
R
J
z
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
j
+
K
J
3
U
f
H
F
w
Q
c
Y
v
0
M
7
>
A
B
n
V
L
S
g
N
E
O
z
G
r
8
d
R
Z
m
W
k
P
T
o
/
C
q
u
2
9
I
y
p
X
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
J
C
/
Z
u
Y
L
2
B
K
F
f
D
H
W
A
E
Q
>
n
c
V
S
g
N
O
y
r
U
9
o
P
q
I
+
X
G
j
7
5
M
m
0
d
T
w
R
3
p
v
k
z
8
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
g
N
c
w
j
n
2
0
K
8
C
D
Q
V
Y
G
f
9
>
A
B
L
S
E
O
y
J
r
U
o
P
I
+
X
7
5
M
m
Z
d
F
T
/
H
q
u
W
3
v
p
z
k
R
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
T
0
H
/
N
8
C
q
n
9
W
y
w
F
A
V
+
U
Q
>
B
c
D
L
S
g
E
O
r
f
Y
o
P
K
I
X
G
j
7
m
R
Z
k
u
p
5
3
2
d
v
M
z
J
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
V
j
w
y
r
L
C
q
m
o
N
O
A
W
7
v
K
Y
u
p
k
B
>
n
c
D
S
g
E
z
G
M
H
8
d
0
Q
F
R
J
2
I
5
+
P
U
f
/
3
X
T
Z
9
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
+
P
3
H
V
5
I
o
9
r
N
7
G
X
>
A
B
n
c
D
L
S
g
E
O
y
f
U
Y
K
q
Q
p
d
z
C
Z
m
k
T
w
/
8
0
W
R
u
2
J
j
M
v
F
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
T
R
U
r
Y
J
m
W
n
B
E
Z
X
g
K
w
>
A
c
V
D
L
S
N
O
y
f
9
o
P
Q
I
+
G
j
7
k
/
8
C
H
0
q
u
p
5
3
2
d
v
M
F
z
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
r
W
Z
0
I
J
o
R
+
c
w
d
2
L
T
u
Y
>
A
B
n
V
D
S
g
N
E
O
y
f
U
9
P
K
j
m
k
p
M
7
8
Q
C
/
F
3
X
z
5
H
v
G
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
0
f
T
k
v
2
G
w
J
I
o
V
R
W
N
d
D
E
g
>
A
B
n
c
L
S
O
y
r
U
Y
9
P
K
Q
+
X
j
7
m
Z
/
8
C
H
q
u
p
5
3
M
F
z
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
9
y
J
+
u
F
r
n
G
R
7
j
z
/
3
N
L
8
>
A
B
c
V
S
g
E
O
f
U
Y
o
P
K
Q
I
X
m
Z
k
T
w
C
H
0
q
p
5
W
2
d
v
M
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
/
F
N
A
7
r
T
q
2
X
8
K
H
L
J
0
S
y
n
P
M
>
B
c
V
D
g
E
O
f
U
Y
9
o
Q
I
+
G
j
R
Z
m
k
w
C
u
d
z
v
p
5
W
3
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
+
P
F
r
J
n
O
8
L
0
5
K
C
U
u
q
M
>
A
B
c
Z
D
S
g
I
3
Q
k
W
w
V
Y
X
2
H
z
G
7
m
/
9
y
T
p
R
E
N
o
v
j
d
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
H
m
T
y
k
v
U
9
5
d
R
S
8
L
P
N
z
c
>
A
B
V
C
7
g
F
0
X
G
+
o
p
Y
W
Q
E
M
Z
w
f
2
O
D
I
K
r
n
/
j
3
u
J
q
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
9
5
o
I
W
M
k
q
g
u
j
V
S
F
>
A
B
H
c
N
8
E
U
v
r
R
2
0
Y
C
D
w
y
z
d
n
3
P
K
O
L
/
m
G
7
Z
+
T
J
p
Q
X
f
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
U
o
k
N
P
A
y
O
H
d
7
B
j
L
2
V
>
9
c
S
w
E
X
z
F
q
J
W
I
Q
Y
u
K
M
g
v
m
5
D
p
C
/
R
T
+
Z
f
n
r
3
8
G
0
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
/
M
2
U
j
I
Y
T
n
k
3
E
J
X
q
w
f
O
u
v
>
A
B
9
H
c
V
C
7
S
g
N
F
r
G
Q
Z
L
5
z
W
m
P
D
K
d
p
R
+
8
0
y
o
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
H
g
2
o
k
c
X
G
F
N
m
7
C
p
P
z
E
V
8
>
A
B
9
S
w
L
q
j
J
W
I
Q
Y
u
K
M
y
v
5
D
d
/
R
T
+
Z
f
n
r
3
O
0
U
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
K
p
Z
R
m
J
S
r
/
W
v
o
k
9
E
c
N
+
>
A
B
H
V
C
7
g
F
2
f
U
u
G
Q
M
L
I
d
Y
D
w
T
3
P
j
n
X
8
q
y
O
0
z
5
DKG-based Pathway
HF-based Pathway
Figure 2: Illustration of the proposed MDNN, consisting of two core pathways: the DKG-based pathway and the HF-based pathway. (1) The DKG-based pathway utilizes the graph neural network to extract the topological structural information and semantic relations from the constructed drug knowledge graph (DKG). (2) The HF-based pathway mines the inter-modality similarities of each heterogeneous feature from multiple sources. (3) The multimodal fusion neural layer is applied to effectively assist the joint representation learning of both the structural information and attribute feature, which explore the cross-modality complementarity of the multimodal data.
Heterogeneous Feature (HF). In this study, heterogeneous features consist of the target feature, substructure feature and enzyme feature. It is expressed as follows:
where Xt ∈ RNd×Nt , Xs ∈ RNd×Ns and Xe ∈ RNd×Ne stand for the target feature matrix, the substructure feature matrix and the enzyme feature matrix, respectively. Nt, Ns and Ne represent the feature number of the targets, the substructures and the enzymes, respectively.
DDI Events Prediction. Given the DDI events matrix Y, drug knowledge graph G and heterogeneous features Xd, we aim to predict specific interaction events between drug di and drug dj . In other words, we formulate DDI events prediction as a multi-class classification problem. Our goal is to learn a prediction function ˆ yij = Γ(di, dj |Θ, Y, G, Xd), where ˆ yij represents the probability of an event between drug di and drug dj , and Θ denotes the model parameters of function Γ.
4 ProposedMethod
Overview. The architecture of MDNN is depicted in Figure 2, which is composed of two main pathways: the DKGbased pathway and the HF-based pathway. The DKG-based pathway utilizes the graph neural network to extract the topological structure information and semantic relations between drugs on the constructed drug knowledge graph. The HF-based pathway aims to extract predictive information from different modalities to enhance the performance of the learned models. The multimodal fusion neural layer is applied to effectively assist the joint representation learning of both the structural information and the heterogeneous feature which explore the cross-modality complementarity of the multimodal data.
In the section, MDNN will be explained in details next, including the DKG-based pathway in subsection 4.1, the HFbased pathway in subsection 4.2 and the multimodal neural fusion layer in subsection 4.3.
4.1 The DKG-based Pathway
We explore the advantage of the abundant information related to the topological structure and semantic relations in the DKG, which is beneficial for DDI events prediction.
Drug Knowledge Graph
For each drug in the DDI matrix, we collect the drug related entities on DrugBank, such as targets, transporters, etc. In order to obtain rich semantic information, we consider the General Function of the tail entities as the relations between the drug and the tail entities. For example, the drug DB05812 has a carrier named serum albumin (Uniprot ID: P02768), and the general function of P02768 is toxic substance binding, leading to the triple of the DKG representation <DB05812, toxic substance binding, P02768>. In this way, we can obtain the drug knowledge graph triples (drug, relation, tail entity) with abundant information including the topological structure and semantic relations.
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
3
O
v
S
w
m
Q
F
r
X
g
7
P
A
o
n
c
>
B
9
H
V
N
E
2
L
z
G
j
Y
u
C
I
U
W
d
J
/
8
Z
D
f
K
5
+
y
T
M
k
p
0
R
q
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
K
W
C
8
Q
7
R
L
E
m
A
M
z
u
S
X
k
>
B
9
H
c
V
w
N
3
o
U
D
J
y
F
g
Y
G
I
5
O
+
p
v
n
f
2
j
/
r
Z
0
T
q
d
P
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
G
K
T
S
Z
j
V
M
/
v
7
J
2
c
m
C
k
g
>
A
B
9
H
w
N
E
3
z
8
o
U
u
D
p
Q
W
d
y
Y
F
5
I
P
L
r
n
X
R
q
f
O
0
+
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
g
B
p
G
Z
8
J
k
7
L
y
j
o
w
M
>
A
9
H
c
V
S
N
E
3
z
U
u
D
R
W
m
v
d
Y
2
F
C
5
I
P
/
K
r
n
X
T
q
Q
f
O
0
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
M
p
8
O
2
Y
+
E
I
S
N
K
F
n
Q
o
>
A
B
H
c
V
C
7
g
J
r
f
W
U
m
w
z
D
k
Z
d
/
9
j
5
0
u
v
L
P
3
y
X
T
R
G
q
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
8
N
M
n
V
H
0
m
J
j
C
Y
R
T
/
+
E
U
>
A
B
c
9
S
w
2
L
X
z
F
q
o
d
I
g
D
f
k
O
3
W
P
G
p
v
y
K
r
Z
7
5
u
Q
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
f
w
9
+
K
E
I
O
G
S
u
m
W
c
n
2
7
o
U
>
A
B
H
V
D
L
N
F
J
r
z
R
X
T
g
j
M
p
Q
Z
C
k
v
8
0
P
d
3
q
/
Y
5
y
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
f
w
9
+
K
E
I
O
G
S
u
m
W
c
n
2
7
o
U
>
A
B
H
V
D
L
N
F
J
r
z
R
X
T
g
j
M
p
Q
Z
C
k
v
8
0
P
d
3
q
/
Y
5
y
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
P
Z
9
K
O
5
3
M
S
V
q
d
X
2
c
n
y
g
>
A
B
7
H
C
N
F
J
r
f
U
u
w
Y
0
m
z
o
k
T
I
R
/
8
v
D
j
W
Q
E
+
G
p
L
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
P
Z
9
K
O
5
3
M
S
V
q
d
X
2
c
n
y
g
>
A
B
7
H
C
N
F
J
r
f
U
u
w
Y
0
m
z
o
k
T
I
R
/
8
v
D
j
W
Q
E
+
G
p
L
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
5
9
n
M
I
O
R
m
D
0
S
z
7
y
>
A
B
/
c
V
L
N
F
J
3
r
P
U
d
u
o
v
Q
g
p
k
H
Z
Y
T
E
f
8
C
j
X
w
q
W
G
2
K
+
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
8
z
r
7
v
R
L
w
D
O
H
A
9
2
k
3
Y
c
V
Q
>
B
/
n
S
N
F
J
U
q
y
M
E
C
G
Z
I
K
W
P
+
f
0
u
o
p
X
j
g
5
m
d
T
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
F
3
B
y
w
c
V
S
g
Z
5
I
K
q
A
U
T
>
+
H
C
7
N
L
0
X
z
G
2
p
M
E
W
Q
D
P
O
m
J
k
u
o
Y
v
n
9
j
d
r
8
f
R
/
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
d
r
Y
L
n
P
O
J
I
M
m
/
C
N
o
y
G
g
>
A
B
+
H
c
V
S
8
E
3
W
u
U
9
k
0
T
v
D
j
z
R
5
w
F
X
p
f
2
7
K
Z
q
Q
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
0
E
f
5
M
Z
V
K
N
g
y
A
v
3
n
o
>
C
B
c
D
L
S
F
J
r
P
U
d
I
j
p
R
G
2
m
k
7
O
q
+
8
H
z
X
/
Y
9
w
Q
T
W
u
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
I
3
v
n
B
k
q
J
d
9
g
2
7
W
N
T
F
O
U
>
A
X
c
V
C
S
L
0
z
G
+
o
p
Y
w
r
m
H
E
M
D
u
Z
8
Q
/
y
j
f
5
R
K
P
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
f
w
9
+
K
E
I
O
G
S
u
m
W
c
n
2
7
o
U
>
A
B
H
V
D
L
N
F
J
r
z
R
X
T
g
j
M
p
Q
Z
C
k
v
8
0
P
d
3
q
/
Y
5
y
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
2
Y
r
9
D
M
E
G
v
o
c
8
O
K
+
0
A
P
>
B
n
Z
L
S
g
F
I
u
3
Q
k
W
w
V
J
X
H
z
U
7
C
m
/
y
T
p
R
N
5
q
j
d
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
y
3
F
U
M
m
W
o
P
w
p
d
E
g
L
+
C
j
u
I
J
c
>
A
B
9
H
V
N
S
8
v
r
R
2
q
k
0
Y
D
z
n
K
O
/
G
7
Z
T
Q
X
5
f
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
c
P
v
q
R
F
r
j
J
X
z
O
7
Y
H
d
u
Z
S
>
A
B
9
V
C
g
N
L
0
G
+
o
p
W
Q
y
E
M
w
T
f
2
k
D
m
I
K
n
/
5
3
U
8
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
d
Q
0
H
X
8
K
c
2
U
V
I
G
M
z
p
f
u
Y
P
k
>
A
B
n
Z
D
L
S
g
F
3
7
N
E
C
o
q
J
5
j
m
+
O
w
/
v
W
r
T
y
9
R
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
0
Q
g
w
E
z
p
T
/
7
f
O
c
J
+
9
k
>
A
B
n
V
D
L
S
N
y
r
U
Y
o
P
K
I
X
G
j
5
M
m
Z
F
H
q
8
3
u
C
v
R
2
W
d
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
U
Y
2
y
G
M
3
Q
X
5
q
D
/
d
T
N
V
I
>
A
B
n
c
9
S
w
E
J
L
z
F
+
R
o
g
7
j
O
Z
f
v
0
p
H
C
W
P
r
8
k
u
m
K
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
V
I
M
E
H
j
r
0
Y
n
5
R
y
g
m
8
>
A
B
c
Z
D
L
S
F
P
v
N
q
d
2
X
J
U
G
Q
7
o
T
O
C
u
f
3
W
w
/
p
z
k
9
K
+
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
y
F
K
S
Q
L
C
T
R
U
Z
X
9
A
v
O
q
m
g
>
B
8
n
c
V
N
E
3
r
j
2
J
P
Y
M
f
z
d
u
k
I
7
H
w
G
W
5
p
o
/
+
0
D
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
/
Z
N
C
w
r
f
I
q
8
X
K
M
c
o
U
5
7
u
>
A
B
n
V
D
L
S
F
J
W
+
d
R
m
k
2
3
T
Q
O
H
9
g
y
j
v
Y
p
G
P
z
E
0
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
c
p
2
9
M
P
H
K
0
J
W
A
d
I
X
Y
j
C
>
B
7
n
V
D
L
S
g
N
E
O
y
r
f
U
o
Q
Z
F
z
m
+
w
3
8
q
k
T
u
G
5
/
R
v
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
o
0
/
W
H
8
Y
R
M
p
m
E
5
B
+
C
y
9
k
>
A
7
n
c
V
D
L
S
g
N
O
r
f
U
P
K
Q
F
j
u
Z
2
T
I
z
w
J
d
X
q
3
G
v
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
I
M
k
p
L
2
N
/
X
m
W
B
8
P
0
v
E
>
A
9
H
c
V
D
S
F
r
f
Z
d
u
o
g
q
R
J
U
n
G
T
3
z
5
C
Y
7
O
j
+
w
K
Q
y
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
S
L
A
p
j
Z
z
Q
O
9
2
o
+
H
K
U
E
d
>
B
c
V
D
N
F
J
3
v
q
k
g
0
w
R
I
C
r
Y
/
T
u
m
y
M
G
7
f
P
n
8
W
5
X
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
M
K
A
Q
g
d
F
j
Y
z
r
L
H
/
>
B
+
X
c
V
D
S
N
J
3
U
2
v
q
E
0
w
R
I
p
C
Z
T
k
m
G
9
y
8
P
7
n
u
o
5
W
f
O
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
7
p
U
q
u
M
3
/
V
n
F
0
X
v
o
z
5
j
I
>
A
B
8
c
D
L
S
N
J
2
k
g
w
Y
m
K
C
r
y
Q
9
W
Z
f
H
R
P
+
E
T
d
G
O
The GNN Layer
The GNN layer is proposed to capture drug topological structure and semantic relations in the drug knowledge graph. The initial representation matrix of the drug knowledge graph G is as follows:
where Nd, Nr and Nk represent the number of drugs, relations and tail entities in the DKG, respectively. e(0) d ∈ Rd , e(0) r ∈ Rd and e(0) t ∈ Rd are served as the initialization of drug embedding, relation embedding and tail entity embedding, respectively, where d is the dimension of embedding in drug knowledge graph.
For each drug di, we uniformly sample a set of a fixed size as Ns(di) instead of using all the neighbors. It is important to explicitly incorporate the semantics of relations into drug representation learning. Thus, we compute the semantics feature score between drug di and tail entity tn with relation rin as follow:
where e(l-1) rin is the relation representation between drug di and tail entity tn after (l - 1)th GNN layer. e(l-1) di is the drug di representation generated from the previous messagepassing steps, memorizing the messages from its (l-1)-hop neighbors. W (p) 1 is the trainable weight matrix, b(p) 1 is the bias vector and p is the number of full connection layers, ⊙ denotes the element-wise product.
Then, we aggregate the messages propagated from the neighborhood Ns(di) to refine the embedding of di. More formally, we first recursively formulate the neighborhood representation of drug di at lth layer. We define the neighborhood aggregation function as:
The final step aggregates the embedding of drug e(l-1) di and its neighborhood embedding e(l) Ns(di) into a vector using the following aggregation function:
where W2 ∈ R(2d)×d is the trainable weight matrix and σ is the activation function ReLU. ⊕ denotes the concatenate operation.
Similarly, we can obtain the representation Edj for drug dj by propagating information from its neighboring nodes. In summary, the advantage of the embedding propagation layer lies in explicitly exploiting the first-order connectivity information for drug representations.
4.2 The HF-based Pathway
In the HF-based pathway, we use heterogeneous features to calculate the drug similarity between DDI events. Each feature corresponds to a set of descriptors, and thus a drug can be represented by a binary feature vector, whose each entry (1 or 0) indicates the presence or absence of the corresponding descriptor. In order to make the drug node representation more dense and improve the accuracy of the vector, we use principal components analysis (PCA) to compress features and reduce the sparsity. We calculate the pairwise drug-drug similarity from feature vectors using the Jaccard similarity measurement.
By using the Jaccard similarity measurement, we can obtain the target similarity matrix Et ∈ RNd×k , substructure similarity matrix Es ∈ RNd×k and enzyme similarity matrix Ee ∈ RNd×k , where Nd stands for the number of drugs, and the superscript k denotes the dimension of heterogeneous feature embedding.
After obtaining the similarity matrix, we can get the embedding of drug di as etdi ∈ Et , esdi ∈ Es and eedi ∈ Ee , respectively.
Finally, to further explore the inter-modal complementarity of the heterogeneous features, we concatenate the three representation vectors as the final heterogeneous features embedding of di, which is formulated as:
Similarly, the embedding E′dj of drug dj can be obtained.
4.3 Multimodal Neural Fusion Layer
Intuitively, the DKG-based and the HF-based pathways provide complementary information to each other. To achieve the best utilization of the information of these two pathways, we consider their coherence and complementarity together in the so-called multimodal neural fusion layer. After obtaining the embedding Edi and E′di for drug di, these embedding are linked together as the final multimodal embedding ˆEdi of drug di. The equation can be described as:
As such, the embedding of di contains not only the heterogeneous feature information, but also the semantic information of the relation and its structural information. Similarly, the final embedding ˆEdj of drug dj can be obtained.
Then, themultimodal fusion embedding ˆEdij withmultiple fully connected layers is used to predict the DDI events:
where W (q) 3 is the trainable weight matrix, and b(q) 3 is the bias vector, q is the number of the full connected layers. ρ denotes the activation function softmax. Finally, we use softmax function and obtain the final prediction score ˆ yij .
For model optimization, we add batch normalization layers to accelerate the convergence, and add dropout layers to avoid over-fitting and enhance generalization ability. And we adopt cross-entropy as the loss function, and empirically train and optimize the MDNN model. In addition, we use L2 regularization to prevent over-fitting of our model.
| Methods | Acc | AUPR | AUC | F1 | Pre | Rec |
|---|---|---|---|---|---|---|
| Logistic Regression | 0.7920 | 0.8400 | 0.9960 | 0.5948 | 0.7437 | 0.5236 |
| K-Nearest Neighbour | 0.7214 | 0.7716 | 0.9813 | 0.4831 | 0.7174 | 0.4081 |
| Random Forest | 0.7775 | 0.8349 | 0.9956 | 0.5936 | 0.7893 | 0.5161 |
| Deep Neural Network | 0.8797 | 0.9134 | 0.9963 | 0.7223 | 0.8047 | 0.7027 |
| DeepDDI [Ryu et al., 2018] | 0.8371 | 0.8899 | 0.9961 | 0.6848 | 0.7275 | 0.6611 |
| DDIMDL [Deng et al., 2020] | 0.8852 | 0.9208 | 0.9976 | 0.7585 | 0.8471 | 0.7182 |
| MDNN | 0.9175 | 0.9668 | 0.9984 | 0.8301 | 0.8622 | 0.8202 |
5 Experiments
In this section, we describe the experimental setups and the results of the performance evaluation on our proposed model in DDI events prediction.
5.1 Experimental Setup
Dataset. In order to demonstrate the effectiveness of our proposed model, we conduct extensive experiments on a realworld dataset including three parts: (1) DDI Matrix: We obtain the verified DDI events data from DDIMDL 1 , which contains 572 drugs and 65 types of events. According to statistics, a total of 37,264 drug pairs have definite drug-drug interactions. (2) Drug Knowledge Graph: According to DDI events, we collect the drug knowledge graph from the DrugBank 2 (version 5.1.7). It is a real-world dataset that contains 572 drugs and 1,614 entities with 76,871 triplets and 157 relations between drugs and tail nodes. (3) Heterogeneous Features: Heterogeneous features released by DDIMDL [Deng et al., 2020] include 1,162 target features, 583 substructure features and 202 enzyme features.
Baselines. In this study, we compare our model against the following baselines, including the traditional and the recent state-of-the-art methods. DDIMDL [Deng et al., 2020] adopts a joint deep neural network framework to learn the representations of drug-drug pairs and predict DDI events. DeepDDI [Ryu et al., 2018] develops a deep learning-based method that reduces the dimension of drug features based on a principal component analysis. We consider several traditional classification approaches, i.e., random forest, k-nearest neighbour, logistic regression and deep neural network.
Evaluation Metrics. We evaluate the prediction performance using several multi-class classification evaluation metrics, including accuracy (Acc), area under the precision-recall-curve (AUPR), area under the ROC curve (AUC), F1 score (F1), Precision (Pre) and Recall (Rec). We use micro metrics for AUPR and AUC, while macro metrics for F1 score, Precision and Recall.
Parameter and Evaluation Settings. The maximum iteration number is set to 100, and we use a batch size of 1,024 and adopt Adam algorithm with a learning rate of 0.001 to optimize all trainable parameters through a random search in each iteration. We set the Ns = 6, l = 1, L2 weight = 1e-8, p = 2, q = 3, dimension d = 128 and k = 256.
1 https://github.com/YifanDengWHU/DDIMDL
To comprehensively evaluate our proposed method, we adopt 5-fold cross validation and randomly divide all DDI pairs into five subsets in our experiments. The evaluation score is the average of the output of the five rounds. We use the early-stopping strategy to prevent over-fitting which automatically stops the training if no improvement is observed after 10 epochs.
5.2 Results and Analysis
In this section, we report the performance of our model and all baselines in Table 1. From the result, we find that our model achieves the best performance in DDI events prediction on the real-world dataset. Particularly, our proposed model outperforms DDIMDL by 3.23% on Acc, 4.6% on AUPR, 0.08% on AUC, 7.16% on F1, 1.51% on Pre and 10.2% on Rec. The better performance of our model is attributed to the fact that our model explores both the drug topological embedding representations in the drug knowledge graph and the cross-modality embedding representations of the multimodal data. Moreover, the comparative study with other state-of-art methods demonstrates that our model achieves the most stable performance which may be due to (a) MDNN incorporates a GNN model to exploit the topological structure information and semantic relations in the drug knowledge graph; (b) MDNN leverages the crossmodality complementary information of the multimodal data. To sum up, it is a preferable achievement in terms of DDI events prediction.
5.3 Ablation Study
To explore how the DKG-based and the HF-based pathways improve the performance of the proposed model, we conduct the ablation study on the following variants of MDNN. MDNNdkg is the model variant where we only consider the topological structures and semantic relations to learn the embedding of drug-drug pairs from the DKG. MDNNhf is the model variant where we only explore cross-modality embedding of drug-drug pairs using only the heterogeneous features. Moreover, as MDNNhf only considers multimodal attribute feature of drug-drug pairs, it performs worse than MDNN and MDNNdkg in all metrics. Figure 3 shows that the ablation results which verify the contribution of each pathway in our model, showing that combining the topological representations in neighborhood with the semantic relations from the DKG and the heterogeneous features is beneficial to improving the DDI events prediction performance. It can be seen that from the results that MDNN outperforms both variants in all metrics.
5.4 Parameter Sensitivity Analysis
In this work, there are three essential parameters, which are the size of neighborhood sample Ns, the number of GNN layers l and the dimension of embedding d in DKG. We fixed other parameters when studying the effect of each of them. The results are shown in Figure 4.
Effect of neighborhood size. We vary the size of sampled neighbor Ns to explore the efficacy of MDNN. Figure 4 shows that our model achieves the best performance when Ns = 6. When Ns is too small, the model cannot fully incorporate the structural information, while an large value of Ns makes the model more prone to be misled by noises.
Effect of GNN layers. We investigate the influence of the GNN layer l by varying its value from 1 to 3. We observe that the performance of our model in all the metrics decreases starting from l = 1, as a larger l brings massive noises to the model. This is also in line with our intuition that using nodes with too many hops makes little difference when encoding the topological information of each drug in DKG. The experiment results implies that l = 1 is often ideal for real cases.
Effect of embedding dimension. In addition, we examine the influence of embedding dimension d by varying from its value from 32 to 512. Intuitively, the performance can be enhanced with a proper d that can encode enough information of drugs and entities from the DKG. When d is too large, however, the model will be affected by the over-fitting.
5.5 Multi-task Analysis
We created two different tasks by randomly splitting the drugs involved into five subsets and using four of them as the training drug set while the remaining one as the test drug set to evaluate the effectiveness of our model. For task A, prediction models are constructed on the DDI between training drugs, and then make predictions for DDI events between training drugs and test drugs. For task B, it is different from
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
V
U
7
Q
X
9
B
u
p
n
k
O
D
f
v
H
C
G
3
0
>
A
c
Z
J
S
g
N
E
I
r
j
F
8
Y
o
z
T
M
y
R
+
/
P
W
d
m
L
2
w
5
K
q
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
p
5
o
Q
+
G
v
A
2
Y
c
C
r
T
8
7
y
m
9
O
S
>
B
H
V
g
N
F
L
0
X
z
w
D
K
k
d
U
M
u
f
J
3
P
j
q
E
I
n
/
W
Z
R
Ns
| Task | Methods | Acc | AUPR | F1 | Rec |
|---|---|---|---|---|---|
| Task A | DNN | 0.6239 | 0.6361 | 0.2997 | 0.2840 |
| DeepDDI | 0.5774 | 0.5594 | 0.3416 | 0.3890 | |
| DDIMDL | 0.6415 | 0.6558 | 0.4460 | 0.4319 | |
| MDNN | 0.6495 | 0.6661 | 0.4471 | 0.4611 | |
| Task B | DNN | 0.4087 | 0.3776 | 0.1152 | 0.1093 |
| DeepDDI | 0.3602 | 0.2781 | 0.1373 | 0.1450 | |
| DDIMDL | 0.4075 | 0.3635 | 0.1590 | 0.1452 | |
| MDNN | 0.4575 | 0.4215 | 0.1697 | 0.1709 |
Table 2: Performance comparison of MDNN with other methods on two different tasks.
task A that MDNN make predictions for DDI events between test drugs.
It can be learned from Table 2 that the experimental results of our model in both tasks are better than other methods. This effectively shows that whether it is between known drugs or new drugs, the utilization of structural information and heterogeneous features improves the prediction accuracy of DDI events, and provides a strong, reliable support for research on DDI events prediction.
6 Conclusion
In this paper, we propose a new MDNN model for drug-drug interaction events prediction. MDNN effectively exploits both the topological information and the semantic relations by leveraging a graph neural network on the drug knowledge graph. Moreover, MDNN also exploits the joint representation learning of both the structure information and the heterogeneous features, which effectively explores the cross-modal complementarity of the multimodal data. The experimental results show that MDNN outperforms the classic and stateof-the-art DDI events prediction models.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (61873288), Zhejiang Lab (2019KE0AB01, 111007-PI2001), Zhejiang Provincial Natural Science Foundation (LZ21F030001), Hunan Provincial Innovation Foundation for Postgraduate (CX20200108) and the Fundamental Research Funds for the Central Universities of Central South University (2021zzts0740).
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
O
D
q
r
P
C
S
j
n
Z
B
H
o
V
7
K
U
>
A
c
J
g
N
E
I
F
p
8
z
Q
9
T
M
k
0
G
+
/
d
v
X
R
5
W
u
Y
2
3
m
y
L
w
f
@inproceedings{ijcai2021p487,
title = {MDNN: A Multimodal Deep Neural Network for Predicting Drug-Drug Interaction Events},
author = {Lyu, Tengfei and Gao, Jianliang and Tian, Ling and Li, Zhao and Zhang, Peng and Zhang, Ji},
booktitle = {Proceedings of the Thirtieth International Joint Conference on
Artificial Intelligence, {IJCAI-21}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Zhi-Hua Zhou},
pages = {3536--3542},
year = {2021},
month = {8},
note = {Main Track},
doi = {10.24963/ijcai.2021/487},
url = {https://doi.org/10.24963/ijcai.2021/487},
}
TY - CPAPER TI - MDNN: A Multimodal Deep Neural Network for Predicting Drug-Drug Interaction Events AU - Lyu, Tengfei AU - Gao, Jianliang AU - Tian, Ling AU - Li, Zhao AU - Zhang, Peng AU - Zhang, Ji T2 - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2021 SP - 3536 EP - 3542 DO - 10.24963/ijcai.2021/487 UR - https://doi.org/10.24963/ijcai.2021/487 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
