Musical Composition Style Transfer via Disentangled Timbre Representations
Musical Composition Style Transfer via Disentangled Timbre Representations
Yun-Ning Hung, I-Tung Chiang, Yi-An Chen, Yi-Hsuan Yang
Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
Main track. Pages 4697-4703.
https://doi.org/10.24963/ijcai.2019/652
Music creation involves not only composing the different parts (e.g., melody, chords) of a musical work but also arranging/selecting the instruments to play the different parts. While the former has received increasing attention, the latter has not been much investigated. This paper presents, to the best
of our knowledge, the first deep learning models for rearranging music of arbitrary genres. Specifically, we build encoders and decoders that take a
piece of polyphonic musical audio as input, and predict as output its musical score. We investigate disentanglement techniques such as adversarial
training to separate latent factors that are related to the musical content (pitch) of different parts of the piece, and that are related to the instrumentation
(timbre) of the parts per short-time segment. By disentangling pitch and timbre, our models have an idea of how each piece was composed and arranged. Moreover, the models can realize “composition style transfer” by rearranging a musical piece without much affecting its pitch content. We
validate the effectiveness of the models by experiments on instrument activity detection and composition style transfer. To facilitate follow-up research,
we open source our code at https://github.com/biboamy/instrument-disentangle.
Keywords:
Multidisciplinary Topics and Applications: Art and Music
Multidisciplinary Topics and Applications: Information Retrieval
Machine Learning: Deep Learning
Machine Learning: Transfer, Adaptation, Multi-task Learning
Machine Learning: Adversarial Machine Learning