Q&A: Query-Based Representation Learning for Multi-Track Symbolic Music re-Arrangement

Q&A: Query-Based Representation Learning for Multi-Track Symbolic Music re-Arrangement

Jingwei Zhao, Gus Xia, Ye Wang

Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence
AI and Arts. Pages 5878-5886. https://doi.org/10.24963/ijcai.2023/652

Music rearrangement is a common music practice of reconstructing and reconceptualizing a piece using new composition or instrumentation styles, which is also an important task of automatic music generation. Existing studies typically model the mapping from a source piece to a target piece via supervised learning. In this paper, we tackle rearrangement problems via self-supervised learning, in which the mapping styles can be regarded as conditions and controlled in a flexible way. Specifically, we are inspired by the representation disentanglement idea and propose Q&A, a query-based algorithm for multi-track music rearrangement under an encoder-decoder framework. Q&A learns both a content representation from the mixture and function (style) representations from each individual track, while the latter queries the former in order to rearrange a new piece. Our current model focuses on popular music and provides a controllable pathway to four scenarios: 1) re-instrumentation, 2) piano cover generation, 3) orchestration, and 4) voice separation. Experiments show that our query system achieves high-quality rearrangement results with delicate multi-track structures, significantly outperforming the baselines.
Keywords:
Application domains: Music and sound
Methods and resources: Machine learning, deep learning, neural models, reinforcement learning
Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI