Tradformer: A Transformer Model of Traditional Music Transcriptions

1 Introduction

Since the introduction of transformers [Vaswani et al., 2017], the state of the art of numerous applications of machine learning has advanced, from natural language processing [Radford et al., 2019; Devlin et al., 2018] to computer vision [Dosovitskiy et al., 2020]. At the core of a transformer is attention: a dynamic and highly parallelizable mechanism by which the model predicts output from previous input. A transformer can potentially learn dependencies over very long ranges, addressing one of the biggest limiting factors of recurrent neural networks like long short-term memory (LSTM) [Casini et al., 2018]. For music sequences exhibiting repetition, a transformer can learn to refer back to what it has already generated, attending to particular material to make inferences. The ability to repeat material over long time scales is seen in the outputs of Music Transformer [Huang et al., 2018] and Musenet [Payne, 2019]. However, transformers have their limits, such as a maximum sequence length and a quadratic memory requirement that scales with it.

The application of attention to modeling transcriptions of traditional music has yet to be thoroughly explored. All entries submitted to The Ai Music Generation Challenge 2020 1 (focusing on Irish double jigs) used either Markov modeling or LSTM [Sturm and Maruri-Aguilar, 2021]. The winning entries were generated by the LSTM-based system folk-rnn (v2) [Sturm et al., 2016]. Brawen (2019) explores tuning the language transformer model GPT-2 on text-based traditional music data; but this model has about 20 times the number of parameters of folk-rnn and little demonstrable gain in quality.

Given attention and its compatibility with the kinds of structures present in Irish and Swedish traditional dance music, a transformer model should excel in generating plausible new tunes in these styles with fewer parameters than the LSTM folk-rnn. Applying the transformer to this task however, was not straightforward. Certain aspects of development had a large influence on performance. To develop a successful transformer model, we performed several iterations of informal comparison tests with folk-rnn models as a benchmark, and also developed visualizations of the internal structure of the transformer, including its use of attention on forming predictions. The final model, which we name Tradformer, has about one-fifth the parameters of folk-rnn and achieves results of similar or better quality for some forms of Irish traditional music. We used transfer learning to adapt Tradformer to a style of Swedish traditional dance, called sl¨angpolska, which is the focus of The Ai Music Generation Challenge 2021. 2 Competing against five others, Tradformer has produced tunes rated the most favorable by five human judges.

In the following, we review the transformer architecture and then present the details of our model. Section 4 discusses numerous decisions we made in engineering Tradformer, including sampling and visualizing the internal components of the model as training proceeds. Section 5 presents how we have used transfer learning to tune Tradformer to adapt to a type of Swedish music. We discuss our results in Sec. 6.

2 Review of a Transformer Model

Let V = {vi : i ∈ I} be a vocabulary indexed by the set I, and a length-N sequence s = (it ∈ I : t ∈ [0,N)) where t is the index into the sequence. The output of a transformer is a categorical probability distribution over I conditioned on s, from which the Nth element S(N) can be inferred. Figure 1 depicts the basic transformer, which operates as follows.

1 https://boblsturm.github.io/aimusic2020/

2 https://github.com/boblsturm/aimusicgenerationchallenge2021

Each element of s is represented by an embedding vector in a de-dimensional real space, summed with a vector encoding its position in the sequence [Vaswani et al., 2017]. More precisely, element t of s is encoded as ws(t) ∈ Rde , and its position is encoded as pt ∈ Rde . Let XN = [x0|x1| . . . |xN-1], where xt = ws(t) + pt. This is processed by the first layer of J attention heads, each operating in a subspace of Rde of dimension dh = de/J (where de is an integer multiple of J). The jth attention head produces query, key and value representations via: Q(j) N = WQ j XN, K(j) N = WK j XN, and V(j) N = WV j XN, where WQ j ,WK j ,WV j ∈ Rdh×de . The jth attention head then forms the scaled inner-product matrix S(j) N = (dh)-1/2 [Q(j) N ]T K(j) N , compresses each column of S(j) N using a softmax operation to form ˆS(j) N ∈ RN×N , and outputs Z(j) N = V(j) N ˆS(j) N ∈ Rdh×N . The outputs of all J attention heads are concatenated ZN = XN + (Z(1) N ,Z(2) N , . . . ,Z(J) N ) ∈ Rde×N , and then passed through a non-linearity Z′N = g(W1ZN), where W1 ∈ R4de×de . Finally, the output of the first layer is XN ← XN + A1Z′N where A1 ∈ Rde×4de . The following attention layer performs the same operation as the first layer, but on the updated XN and with different parameters. After L attention layers, the final layer takes the last column of XN and computes the parameters of the distribution over I by θN = softmax(Σ[XN ]∗N + b) where Σ ∈ R|I|×de and b ∈ R|I| .

Training a transformer entails learning the embedding vectors of the vocabulary, the linear transformations of the query, key, and value in each attention head in each layer, the linear transformations of the concatenations of the results of attention, the weights between each hidden layer, and finally the parameters of the softmax layer. The standard loss function is cross-entropy.

3 Tradformer Architecture and Training

We aim to build a transformer that performs as well as the LSTM folk-rnn (v2) [Sturm et al., 2016], and so use the same data and representation. 3 This dataset comes from thesession.org, and contains 23,636 transcriptions, mainly of traditional Irish dance music, expressed with a vocabulary of 137 tokens representing meter, mode, pitch, etc. We limit the maximum length to 256, removing only 2,362 sequences.

3 See data v2 in https://github.com/IraKorshunova/folk-rnn

We make the dimension of the vocabulary embedding de = 128. Using a dimension larger than the vocabulary adds complexity and training time, and we found no benefit through human evaluation (described more in Sec. 4). We also found worse results using a dimension of 64 (powers of 2 are not necessary but often they allow the hardware to work faster). The positional encoding of Tradformer is that given in [Vaswani et al., 2017], which is based on a combination of sinusoids. This can be learned as well, but our evaluation showed it to work slightly worse. Tradformer has L = 16 layers each with J = 4 attention heads. This was determined by testing numbers of layers ranging from 4 to 16, and numbers of attention heads from 1 to 8.

We experimented with different batch sizes, learning rates and numbers of epochs. The final learning rate is 5.0-4 . The batch size was set to 64 from considerations of our computational hardware. We used in total 50 epochs. These parameters however did not seem to have a major impact on model convergence. When Tradformer showed small to no improvement in its validation loss during training, we could still detect improvement in the quality of generated tunes. An explanation could be that, looking at the loss function, a wrong token is still wrong even if it is musically plausible. As training goes by, the average number of mistakes could be the same but the quality of those mistakes could be improving from a music theory standpoint.

For sampling, Tradformer employs a combination of beam search and nucleus sampling similar to that proposed in [Shaham and Levy, 2021]. Tradformer starts with top-p sampling (only looks at the most likely token up to p percent of the probability mass) and then uses at most k of those as branches to search a tree of depth of maximum depth D. This results in a sample space of at most kD token sequences with a probability distribution given by the softmax of the sum of the logits of the component tokens. This increases computational cost, but the increase in output quality was clear. Tradformer sets D = 3, p = 0.99 and k = 3.

We find by human evaluation that Tradformer performs as well as folk-rnn (v2) in generating tunes in 6/8 and 4/4 meter. The appendix contains the scores and comments for some of these comparisons 4 . Figure 2 shows one example generated by Tradformer. The tune has a very common bipartite structure, with each part demarcated by repeat signs. The parts are very similar and only differ in their first bars - a common structure in Irish traditional dance music. The only odd part is that the ending of each section does not resolve.

4 Engineering Considerations

This section discusses a variety of considerations that guided the engineering of Tradformer.

4.1 Human Evaluation

Many design choices were dictated not only by measuring a difference in loss, but also the quality of model outputs by human evaluation. Periodically, we had Tradformer and folkrnn (v2) each generate 10 tunes in a given meter, ordered them randomly without any identification, and then had a musician competent in Irish traditional music rate each tune as good or bad. This provided a more complete picture of the musical quality of the two models, and revealed when the two models were generating material at a similar level of quality. It was an inexpensive and effective way of gauging progress.

4.2 Sampling

The sampling strategy of Tradformer is extremely important. We found the biggest improvement in the quality of the generated music came from replacing a naive approach with a more sophisticated one based on beam search. Our early models were using a naive sampling approach, where single tokens were drawn from a categorical distribution parametrized with the softmax probabilities. The outputs contained counting errors and drifting melodies, and were rated very poor against melodies generated by folk-rnn.

Lowering the softmax temperature was slightly helpful in preventing such problems, but created too much repetition. Furthermore, temperature is applied equally even when the context makes it unnecessary. We thus tried employing top-k and top-p sampling [Holtzman et al., 2019], where unlikely tokens, according to rank or probability mass, are removed before the remaining probabilities are rescaled by softmax. With p ∈ [0.9, 0.99] we saw fewer mistakes but also less variety. This could be countered by increasing the temperature above 1.0, but we observed the melodies were still drifting. Even a nudge to these hyper-parameters led to wildly different results and finding the right balance was difficult.

This led to our development of the beam search sampling described in Sec. 3. Beam search provided melodies with more direction and virtually no mistakes but, as noticed for NLP [Holtzman et al., 2019], makes for less "surprising" outputs. Correctness and variety seem to be two different objective that are difficult to obtain at the same time. However, leaving some control over those parameter to the user can provide creative opportunities. We can provide further quality control in the output of Tradformer by using rejection sampling: only present a tune to a user output if it meets strict criteria. This is described more in Sec. 5.

4 See https://github.com/mister-magpie/tradformer for the appendix and code

4.3 Parameter Analysis

We found it useful to visually analyze the parameters of the model as we developed Tradformer. This provided a way to see if training was proceeding correctly, and also whether the model was learning relevant concepts from the music transcription sequences. Figure 3 shows an example of this with the tune from figure 2. Near the top we see high probability mass repeated in the measure line tokens. Near the bottom we see recurring patterns in pitch tokens, where confidence increases toward the end of the two phrases. The strange ending of each phrase is due to an unlikely draw from the distribution the first time, but Tradformer then reprises it confidently in the second part. Plotting these parameters reinforce the idea that the model has learned good representations because we see how its predictions narrow as it generates a sequence.

Figure 4 shows the matrix of the last layer Σ, with rows corresponding to tokens, arranged according to token type. Each column of Σ is essentially a prototype shape of the parameter vector θN. Tradformer combines these shapes to deform the bias vector b. We see bands in these shapes where magnitude appears to be scaled with the frequency of the token type in the training dataset. We see two stripes near the top associated with meter and key tokens. The band near the middle denotes the pitch tokens without accidentals.

Figure 5 visualizes the cosine distance between all unique pairs of token embeddings. The strong diagonal line comes from each token being colinear with itself. Several clusters and diagonal lines are clear and correspond to musical function. In particular, the off-diagonal lines relate pitches to their enharmonics or octaves. Such structures were also found in gate parameters of folk-rnn [Sturm, 2018].

Figure 6 shows the activity of attention heads ˆS(j) N in three layers during the generation of the tune in Fig. 2. This can reveal how the attention mechanism is referring to what it has seen to form predictions, but does not provide a complete explanation of the model [Wiegreffe and Pinter, 2019]. In the first layer, we see one head paying constant attention to the key token (exemplified by the vertical line). In the eighth layer a head shifts its attention to the most recent measure token. At the deepest layer, we see an attention head looking at repetitions of melodic material in each section.

5 Irish to Swedish by Transfer Learning

After obtaining satisfactory results in modeling Irish music, we applied transfer learning to tune Tradformer to a particular style of traditional Swedish dance music: sl¨angpolska - the focus of The Ai Music Generation Challenge 2021. Compared to the problem of modeling Irish music, this style presents a difficulty by the limited amount of data available, as well as its unique musical characteristics.

We first scraped tunes from an online resource of Scandinavian traditional music (folkwiki.se) that are tagged as "sl¨angpolska" or had the word in its title. We then filtered incomplete and duplicate tunes, removed irrelevant material, and formed a vocabulary compatible with the Irish music dataset used to train Tradformer. Since sl¨angpolskor have many semiquavers, their transcription is simplified by use of a field denoting the semiquaver as the base duration. In the Irish dataset the base duration is a quaver. We thus add a base duration token to the vocabulary. We remove vocabulary elements that are rare in the Irish dataset, and add an explicit padding token [Roccetti et al., 2019]. The size of the vocabulary thus decreased from 137 to 128. To retain as much data as possible, we set the maximum sequence length to 512. The final training dataset has 593 tunes. We fine-tuned Tradformer for 30 epochs on the sl¨angpolska data as more training could result in overfitting.

We see that the model seems to learn characteristic rhythmic patterns, as well as the harmonic language of sl¨angpolskor (mostly in major and harmonic minor modes). Figure 7 shows one tune output from this model, which is considerably different from the tune shown in Fig. 2.

We observed that the outputs of the tuned model suffered from problems that can be traced back to the lack of data. Most notably, certain patterns tended to be repeated too often. The model had a tendency to get "stuck" in cycles of two bars - a behavior not seen in either the Irish or Swedish datasets. We also found miscounted measures. The Swedishtuned Tradformer would create odd-length sections, but not as often as seen in the Swedish dataset. Furthermore, we found it hard to balance between interesting output and the risk of too much repetition. Hallstr¨om et al. [2019] report that fine-tuning a folk-rnn model on the whole FolkWiki database (not just sl¨angpolska) also presented difficulties in adaptation, and generated melodies described as unfocused.

The evaluation procedure in The Ai Music Generation Challenge 2021 motivated us to employ rejection sampling to filter out generated material that does not meet strict criteria, i.e., improper meter, rhythm and poor structure. We devised a series of conditions and generated new tunes until the model produced the 1000 tunes required in the challenge submission. Inspired from the analysis by Sturm and Ben-Tal [2017], we formed criteria by looking at statistics from the original dataset and choosing reasonable values. One condition was that all bars must have the correct number of notes. Another condition was that a tune does not contain too many repeated measures, and in particular too many repeated couples - as this type of repetition happens frequently in the generated material but not in the training data. We also checked the tune length and pitch range, along with the number of bars in each section. Very short or lengthy sections show a generation that has gone amok. Rejection sampling discarded around one tenth of the outputs, evidencing the baseline quality of the model.

The five human judges of the challenge graded ten randomly selected tunes for each of six participants along four dimensions: danceability, stylistic coherence, formal coherence and playability. Output generated by Tradformer received in total 88 As, 61 Bs, 37 Cs, and only 3 Fs. The next closest model received 58 As, 67 Bs, 55 Cs, and only 8 Fs. The benchmark - which is just folk-rnn fine tuned on FolkWiki [2019] received in total 42 As, 50 Bs, 49 Cs, and 13 Fs. The appendix contains the Tradformer tunes and their scores. The lowest scoring Tradformer outputs all feature excessive repetitiveness, due to rejection sampling being sidestepped by slight variations in each "repeated" measure.

6 Discussion

The starting point for Tradformer was a minimal PyTorch implementation of GPT-2 [Radford et al., 2018]. 5 From there proceeded several iterations of design, testing and analysis. It was important early on to employ methods of evaluation by which we could gauge progress. This included human evaluation of outputs as well as visualization of the internals of the model. The inspection of loss plots gives only a limited picture of the success of the system. We found that musical quality improved significantly after carefully considering how we sample from the model. The resulting model, Tradformer, appears equal in quality to the LSTM folk-rnn, but using fewer parameters (3,205,504 compared to 5,599,881). Furthermore, the fine-tuning of Tradformer on a Swedish music dataset appears to have been more successful than the fine-tuning of folk-rnn. One limitation of Tradformer that folk-rnn does not have is that the former is limited to sequences of a maximum length. This can be addressed in future work.

5 https://github.com/karpathy/minGPT

The application of machine learning to modeling music sequences is motivated by many reasons [Pearce et al., 2002], not the least of which is augmenting the ways in which music can be created. Toward this end, a measure of usefulness of such systems is how much they motivate or inhibit music creation. As an example, the tune in Fig. 7 is interesting and enjoyable, and in the neighborhood of a good tune. We titled this piece Ugglas Polska (Swedish for "Owl's Polka"), and performed it on our instruments, after agreeing on a number of changes to make it into a more musically appealing piece. 6 Figure 8 shows the tune after making some changes: transposing from C minor to B minor (more appropriate key for the instruments we play); introduction of some tonal ambiguity through the use of D#; bar 6 slightly altered to include an A#; bar 7 made to outline the dominant chord resolving in bar 8; bars 11, 13 and 14 replaced to develop and vary the pattern presented in bars 9 and 10; bar 12 transposed down an octave in order to avoid jumps and continue the phrasing; bars 15 and 16 rewritten to revisit the ending of the first part to create a satisfying conclusion. Such adjustments to generated material, which are not always necessary fixes but rather aesthetic decisions, can be part of the workflow of a composer.

7 Conclusion

This paper introduces Tradformer: a transformer-based architecture applied to modeling traditional dance music from Ireland and Sweden. Particular focus is given to the variety of engineering decisions that considerably improved performance, and provide reusable insights for modeling sequences. Namely visualization combined with human expertise prove fundamental in evaluating generative models, and provide a richer view on model performance than metrics like crossentropy. Periodically visualizing the internal structure of the model provides useful information, sometimes confirming whether the model is learning relevant concepts, and sometimes motivating the refinement of the training data and sampling strategies. Our development of Tradformer also highlights how fundamental the role of human expertise is when designing machine learning models. The complexity of the task and datasets, together with the lack of strong theoretical boundaries for deep neural networks (at least at the moment) make invaluable the intuition that a domain expert brings to the table. Since measuring music quality with respect to a corpus is not clear save for a comparison of basic descriptive statistics, the input from an expert can shed light on whether one is making progress.

6 https://bit.ly/ugglas polska

Acknowledgments

This work was supported in part by the grant ERC-2019-COG No. 864189 MUSAiC: Music at the Frontiers of Artificial Creativity and Criticism. Thanks to Nicolas, Laura, Marco and Joris.