Curriculum-Based Self-Training Makes Better Few-Shot Learners for Data-to-Text Generation

1 Introduction

Recently, text-to-text pre-trained models like BART [Lewis et al., 2020] and T5 [Raffel et al., 2020] have emerged in the field of natural language generation (NLG). Their main idea is to capture the relationship among texts by reconstructing original sentences with the input of corrupted sentences. These models can achieve state-of-the-art performance in various NLG tasks via fine-tuning on downstream datasets.

Despite the success of text-to-text pre-trained models, it's challenging to directly apply them to few-shot NLG tasks because their performance can be largely restricted by the number of labeled data [Chen et al., 2020a; Chen et al., 2020b]. Especially in the task of few-shot data-to-text generation, it's hard to learn the complex relationship between source structured data and target texts via limited labeled data [Gong et al., 2020]. Existing works commonly continue pre-training on large amounts of structured data without corresponding texts for task adaption, aiming to enhance the model performance [Gururangan et al., 2020]. As shown in Figure 1(a), this task-adaptive pre-training strategy appears between general pre-training and fine-tuning, whose purpose is to adapt general pre-trained models (such as BART) to specific tasks (such as data-to-text generation) using unlabeled task data [Xing and Wan, 2021]. We argue that task-adaptive pretraining only captures the relationship among structured data but fails to explicitly model the alignment between structured data and texts, thereby restricting the model performance on the few-shot data-to-text generation tasks.

∗Corresponding author

In this paper, we introduce self-training [Scudder, 1965] as a better few-shot learner for data-to-text generation. As shown in Figure 1(b), self-training is a teacher-student framework where the teacher model creates synthetic labels for the unlabeled data, and the student model is trained on the pseudo-labeled data constructed by the teacher model. Compared with task-adaptive pre-training, self-training explicitly models the relationship between structured data and texts by training the student model on the pseudo-labeled data generated by the teacher model, instead of solely conducting unsupervised pre-training on the unlabeled structured data. We argue that the main challenge falls into the quality of pseudolabeled data. Even if the teacher model is initialized with pre-trained models, it may generate low-quality texts especially when dealing with the structured data with complex topologies, which hurt the performance of the student model.

Thus, we present a novel method called Curriculum-Based Self-Training (CBST) to address the challenge. This method utilizes curriculum learning [Bengio et al., 2009] to construct pseudo-labeled data from easy cases to hard ones, and leverages such data into the training process at different iterations. Specifically, we divide the unlabeled dataset into different subsets based on some difficulty metric such as the number of input triples [Ribeiro et al., 2020b]. At each iteration, we first collect the unlabeled data that satisfy the difficulty metric of the current curriculum. Then, we generate synthetic texts for these data using the teacher model, and select the pseudo-labeled data based on the coverage and generation probability. Finally, we train the student model on the labeled data and the selected pseudo-labeled data, and make it act as the teacher model at the next iteration. This method is expected to gradually increase the difficulty of generating texts for unlabeled structured data and improve the quality of pseudo-labeled data constructed by the teacher model.

Our contributions are mainly as follows 1 :

• We introduce self-training to improve text-to-text pretraining for few-shot data-to-text generation. This method explicitly captures the relationship between structured data and texts by generating texts for unlabeled structured data with the teacher model and training the student model on the pseudo-labeled data.
• We propose a novel method called CBST to alleviate the side-effect of low-quality pseudo-labeled data, which introduces curriculum learning to effectively leverage the unlabeled structured data into the training process in the order determined by the difficulty metric.
• We conduct extensive experiments in the few-shot setting of WebNLG and WikiBio datasets. Results show that our method can outperform fine-tuning and taskadaptive pre-training methods, and achieve state-of-theart performance on these two benchmarks.

2 RelatedWork

2.1 Data-to-Text Generation

Early studies on data-to-text generation mainly focus on how to encode the structural information of input data better. Thus, researchers devise complex encoder structures based on sequential neural networks [Trisedya et al., 2018] and graph neural networks [Ribeiro et al., 2020b] to achieve this goal. Recently, since text-to-text pre-trained models have shown promising performance in various NLG tasks, some works directly fine-tune these pre-trained models including BART [Lewis et al., 2020] and T5 [Raffel et al., 2020] on data-to-text datasets and report impressive results [Ribeiro et al., 2020a; Kale and Rastogi, 2020]. Other works adapt pre-trained models to data-to-text generation tasks by designing specific pre-training tasks such as reconstructing structured data and texts, which further improve the model performance in both supervised and few-shot settings of downstream datasets [Chen et al., 2020a; Ke et al., 2021; Xing and Wan, 2021].

1 The codes are available at https://github.com/kepei1106/CBST.

Compared with existing works, we introduce self-training to explicitly capture the relationship between structured data and texts, instead of solely relying on unsupervised taskadaptive pre-training. Our method is expected to utilize unlabeled data more effectively and further improve text-to-text pre-trained models in few-shot data-to-text generation.

2.2 Self-Training

Self-training [Scudder, 1965] is a teacher-student framework to leverage unlabeled data through semi-supervised learning. Self-training has been applied to the tasks of text classification [Du et al., 2021] and generation [He et al., 2020]. With the development of pre-trained models, recent works also show that self-training is complementary to pre-training for various classification tasks [Du et al., 2021].

For comparison, our work is the first attempt to utilize selftraining to improve text-to-text pre-trained models for fewshot data-to-text generation. Moreover, we consider the difficulty of generating texts for different structured data and introduce curriculum learning [Bengio et al., 2009] to incorporate unlabeled data into the self-training process at different iterations to improve the quality of pseudo-labeled data.

3 Method

3.1 Task Definition andModel Overview

Given the text-to-text pre-trained model fθT2T , the downstream labeled dataset DL = {(GLi ,XL i )}mi=1 , and the unlabeled dataset DU = {GUi }Mi=1 where GLi / GUi denotes the structured data and XL i indicates the annotated texts (M ≫ m), our goal is to obtain a data-to-text pre-trained model which can perform well in the few-shot setting of downstream datasets.

The overview of our proposed method is shown in Figure 2. We follow the existing works to linearize the structured data as a sequence of triples containing subjects, predicates, and objects [Ribeiro et al., 2020a; Chen et al., 2020a] as shown in Figure 3. We first initialize the teacher model fθT with the pre-trained model fθT2T and train it on the labeled dataset DL. Then, we train the teacher model and the student model iteratively. At each iteration, the teacher model fθT generates synthetic texts for the unlabeled dataset D′U which contains the data satisfying the difficulty metric of the current curriculum. Then we select the pseudo-labeled dataset DPL based on the coverage and generation probability of the teacher model. The student model fθS which is also initialized with Initialization Text-to-Text PLM (𝑓𝜃೅2೅ )

Initialization

Text-to-Text

PLM (𝑓𝜃೅2೅ )

Unlabeled Dataset

( 𝐷𝑈 = 𝐷𝑈ሺ1ሻ ∪ ⋯ ∪ 𝐷𝑈ሺெ𝑐ሻ )

Curriculum Segmentation

Unlabeled Dataset (𝐷𝑈 )

Figure 2: Overview of CBST. Dotted lines indicate that the teacher is initialized with the text-to-text pre-trained model and trained on the

labeled dataset only before the first iteration. Afterward, the teacher model is initialized by the student model from the previous iteration.

MC denotes the number of curriculum, and D′U contains the unlabeled data included in the current curriculum at each iteration t.

Algorithm 1 Curriculum-Based Self-Training (CBST)

Require:

Labeled dataset: DL = {(GLi ,XL i )}mi=1

Unlabeled dataset: DU = {GUi }Mi=1

Curriculum segmentation criterion: C

Text-to-text pre-trained model: fθT2T

Ensure:

Data-to-text pre-trained model: fθD2T

• 1: Use C to split DU into D(1) U ,D(2) U , ...,D(MC) U
• 2: Initialize the teacher model fθT with fθT2T and train fθT on DL via Eq.(1)
• 3: for t = 1, 2, · · · ,MC do
• 4: D′U ← ⋃ti=1 D(i) U
• 5: Apply fθT to the unlabeled dataset D′U to acquire the generated texts via Eq.(3)
• 6: Select the pseudo-labeled data based on coverage and generation probability to construct the dataset DPL
• 7: Initialize the student model fθS with fθT2T and train fθS on DPL ∪ DL with input noise via Eq.(4)
• 8: fθT ← fθS
• 9: end for

• 10: fθD2T ← fθS
• 11: return fθD2T

At each iteration, we first utilize the teacher model to generate the synthetic texts for the unlabeled dataset D′U ⊂ DU which includes the unlabeled structured data that satisfy the difficulty metric of the current curriculum:

Then, we select the generated texts based on coverage and generation probability which reflect the generation quality [Gehrmann et al., 2018] to construct the pseudo-labeled dataset DPL with M ′ samples. Specifically, we choose the pseudo-labeled data where the proportion of subjects and objects appearing in the generated text is larger than ϵcov and the generation probability of the text ranks top-ϵgen to construct the dataset for the student model training. Here, ϵcov and ϵgen are both hyper-parameters.

Student Model Training

The goal of this module is to train the student model on the pseudo-labeled dataset and the labeled dataset. Initialized with the text-to-text pre-trained model fθT2T , the student

the pre-trained model fθT2T is trained on the pseudo-labeled dataset DPL and the labeled dataset DL. Finally, the student model fθS acts as the teacher model fθT in the next iteration.

3.2 Curriculum-Based Self-Training

Our proposed method is shown in Algorithm 1, which consists of three main steps: curriculum segmentation, pseudolabeled data construction, and student model training.

Curriculum Segmentation

Curriculum segmentation is designed based on the difficulty of generating texts for structured data. In this paper, we choose the number of triples as the difficulty metric to segment the unlabeled dataset DU into MC subsets. This metric reflects the complexity of structures in the input data and has a significant impact on the quality of the generated texts [Ribeiro et al., 2020b]. We also try other metrics in §4.6.

Pseudo-Labeled Data Construction

This module aims to generate pseudo-labeled data with the teacher model at each iteration. Prior to the first iteration, the teacher model fθT is initialized with the pre-trained model fθT2T and trained on the labeled dataset DL with the following loss function:

where l(X,G; fθ) indicates the negative log-likelihood loss and is commonly used in the sequence generation tasks:

Note that except for the first iteration, the teacher model is initialized by the student model from the previous iteration.

model fθS is trained with the following loss function:

where NF(G) denotes the noise function perturbing the structured data G. Existing works show that input noise is beneficial for self-training because it increases the smoothness of the student model [He et al., 2020]. Inspired by the existing works [Jia et al., 2019; Hoyle et al., 2021], we devise two-level noise functions as shown in Table 1, including word substitution at the semantic level and triple reordering at the structural level. Specifically, we substitute each word by its synonym with the probability pword and shuffle the order of triples in each sample with the probability ptriple, where pword and ptriple are hyper-parameters. These functions are expected to perturb the input structured data while largely keeping the semantic and structural information.

In practice, we adopt a separate training strategy which shows better performance than joint training [He et al., 2020]. This strategy first trains the student model on the pseudolabeled dataset DPL with the first term of LS, and then trains it on the labeled dataset DL with the second term.

4 Experiment

4.1 Dataset

WebNLG. This dataset aims to generate textual descriptions for RDF triples [Shimorina and Gardent, 2018]. The number of instances in training / validation / test set is 34,352 / 4,316 / 4,224, respectively. We followed the existing works [Chen et al., 2020a] to pre-process this dataset and use 0.5%, 1%, 5%, 10% of the training instances as the labeled datasets in the few-shot setting.

WikiBio. This dataset aims to generate the first sentence of biography descriptions for Wikipedia tables [Lebret et al., 2016]. The original split of the training / validation / test set is 582,658 / 2,000 / 72,831. We followed the existing works [Chen et al., 2020a; Chen et al., 2020b] to pre-process this dataset and use 50, 100, 200, and 500 samples from the training dataset as the labeled datasets in the few-shot setting.

We further constructed the unlabeled dataset for each benchmark dataset based on GenWiki [Jin et al., 2020]. This dataset consists of general-domain unpaired structured data and texts sharing the same content distribution. We directly removed the texts in this dataset, and filtered out the structured data that do not have the subjects and objects appearing in the corresponding labeled dataset. Thus, we obtained two unlabeled datasets for WebNLG and WikiBio, respectively, which contain 375,408 / 163,804 samples of structured data without annotated texts 2 .

4.2 Implementation Detail

In our self-training algorithm, we set the number of curriculum MC to be 3. For both WebNLG and WikiBio datasets, we split the corresponding unlabeled dataset into 3 subsets which contain the structured data with (≤2) / 3-4 / (≥5) triples. For the hyper-parameters to select pseudo-labeled data, we set ϵcov = 1.0, ϵgen = 50%. The probabilities of word substitution and triple reordering were set to pword = ptriple = 0.4.

As for the model structure, we used BART [Lewis et al., 2020] as the text-to-text pre-trained model in our experiments. The base version of BART was adopted because of the limited computational resources. We followed BART to use Byte-Pair Encoding vocabulary with the size of 50,265. The training epoch at each iteration was set to be 20. The learning rate was 0.00003. The batch size was 32 / 24 for WebNLG / WikiBio, respectively. The maximum length of linearized structured data was 256 / 384 for WebNLG / WikiBio, respectively, while the length of text sequences was 128.

4.3 Baseline

Direct Fine-Tuning. This category of baselines directly fine-tunes the state-of-the-art pre-trained models including KGPT [Chen et al., 2020a], Switch-GPT-2 [Chen et al., 2020b] and BART [Lewis et al., 2020] on the labeled data without the use of unlabeled data. We denoted these baselines as FT-KGPT, FT-Switch-GPT-2 and FT-BART, respectively.

Task-Adaptive Pre-Training. This category of baselines designs task-adaptive pre-training methods on unlabeled data before fine-tuning. We chose two representative pretraining tasks: 1) Sequence-level reconstruction (SeqRecon) from BART [Lewis et al., 2020] which decodes complete linearized structured data when encoding corrupted structured data; 2) Graph-level reconstruction (GraphRecon) from JointGT [Ke et al., 2021] which predicts the masked tokens at the output layer of the encoder with the input of corrupted structured data. We denoted them as TAPT-SeqRecon and TAPT-GraphRecon, respectively.

For a fair comparison, we also used BART [Lewis et al., 2020] as the text-to-text pre-trained model for task-adaptive pre-training baselines. We presented the results of our model CBST with two ablation models, i.e., CBST w/o CL and CBST w/o CL & Noise. The former ablation model removes curriculum learning from CBST, which is similar to noisy self-training [He et al., 2020]. The latter one simultaneously removes curriculum learning and input noise, which acts as vanilla self-training. All the results were presented with the average values over 3 runs.

2 We have conducted triple-level matching on the filtered GenWiki and the test set of WebNLG / WikiBio. The results show that there is no overlap between them.

4.4 Automatic Result

We followed the existing works [Shimorina and Gardent, 2018; Chen et al., 2020a] to adopt BLEU-4 [Papineni et al., 2002], METEOR [Banerjee and Lavie, 2005], and ROUGEL [Lin, 2004] to evaluate the generated results on WebNLG, and use BLEU-4 as the metric on WikiBio.

The main results on WebNLG and WikiBio are shown in Table 2 and 3. We can observe that CBST significantly outperforms fine-tuning and task-adaptive pre-training methods in all the settings, which shows that our method can explicitly learn the relationship between structured data and texts via self-training and improve the model performance. The comparison between CBST and the ablation models indicates that curriculum learning can alleviate the problem of lowquality pseudo-labeled data and further improve the performance. The noise functions also contribute to the final performance by increasing the smoothness of our model.

4.5 Human Evaluation

To further evaluate the quality of generated results, we conducted human evaluation in the 1% setting of WebNLG. We followed the existing works [Ribeiro et al., 2020b] to use two criteria: fluency (whether a sentence is grammatically fluent)

and adequacy (whether a sentence clearly describes the structured data). We randomly sampled 100 structured data from the test set, and collected the generated results from CBST and other baselines. We adopted pairwise comparison [Ke et al., 2021] between CBST and other baselines. For each pair of generated texts (one from CBST and the other from the corresponding baseline, given the same input structured data), three annotators were hired to determine which text is better (i.e., win, lose or tie) in terms of the above metrics.

Results in Table 4 show that CBST can significantly outperform the baselines based on direct fine-tuning and taskadaptive pre-training in both fluency and adequacy. In addition, the improvement of CBST over two ablation models shows the effectiveness of our curriculum learning module and noise functions to generate fluent texts which describe structured data more clearly. We also calculated Fleiss' Kappa [Fleiss, 1971] for each pairwise comparison to measure the agreement among different annotators, where the results in Table 4 show moderate agreement (0.4 ≤ κ ≤ 0.6).

4.6 Ablation Study

We conducted a detailed ablation test to study the effects of different noise functions, selection criteria, and difficulty metrics. We removed each module respectively and presented the results on WebNLG (1%) in Table 5. Note that w/ DiffLen denotes that we used the length of linearized structured data as the difficulty metric.

Results in Table 5 show that all the modules contribute to the final performance. As for two noise functions, the performance of CBST degrades more in the setting of removing word substitution, which perturbs structured data more flexibly. In terms of the selection criterion, both coverage and generation probability improve the quality of pseudo-labeled data and contribute to the final performance. We can also observe the performance drop when CBST used the length of linearized structured data as the difficulty metric, indicating that the number of input triples is a more proper metric to reflect the difficulty of text generation from structured data.

4.7 Analysis on Pseudo-Labeled Data

To study whether our method can improve the quality of pseudo-labeled data via curriculum learning, we evaluated the quality of pseudo-labeled data generated by the teacher model at the last iteration before selection. We resorted to human evaluation since there is no annotated text for unlabeled structured data. We randomly sampled 100 unlabeled structured data and collected the generated results of the teacher models from CBST and the ablation models at the last iteration. Three annotators were hired to judge each generated text from the following fine-grained aspects: 1) Hallucination: whether the generated text includes non-existing facts; 2) Missing fact: whether the generated text misses input facts; 3) Fluency: the fluency score of generated texts (score 1-5 where 5 indicates fully fluent sentences).

We presented the fluency score and the proportions of generated results that belong to hallucination / missing fact in Table 6. Results show that curriculum learning in our method plays an important role in generating fluent and adequate texts to describe unlabeled structured data, resulting in better model performance. The relatively limited improvement on missing facts may be because our base model BART already has the strong ability to generate texts that appear in the input via its pre-training tasks based on text reconstruction.

4.8 Analysis on CurriculumLearning

To further analyze how curriculum learning helps selftraining in few-shot data-to-text generation, we first demonstrated how the number of curriculum (MC) affects the fi- Table 7: BLEU-4(B-4), METEOR(M), ROUGE-L(R-L) on WebNLG (1%) with different numbers of curriculum.

Iteration

Figure 4: BLEU-4 of CBST and two ablation models at different iterations.

nal performance. The results in Table 7 show that the best performance is reached at MC = 3. When MC is smaller, the teacher model needs to generate texts for unlabeled structured data with multiple triples at early iterations, which may degrade the quality of pseudo-labeled data and the final performance. In contrast, when MC is larger, the student model is trained on easy unlabeled data for many iterations and cannot utilize the hard cases until the late iterations, which may also affect the model performance.

Furthermore, we set MC = 3 and visualized the performance of the student model at each iteration in Figure 4. Note that the values when the number of iterations equals 0 indicate the performance of direct fine-tuning. At early iterations, the two ablation models perform better because they directly incorporate the whole unlabeled dataset into self-training and acquire a larger pseudo-labeled dataset. However, the improvement of their performance is extremely limited at the second and third iterations since the low-quality texts generated by the teacher model at early iterations may restrict the model performance. For comparison, CBST only utilizes the unlabeled data that satisfy the difficulty metric at each iteration to avoid low-quality pseudo-labeled data. Despite the worse performance at early iterations due to the smaller number of unlabeled data included in the self-training process, CBST still obtains the best performance at the last iteration.

5 Conclusion

We introduce self-training to improve text-to-text pre-trained models on few-shot data-to-text generation tasks, which can utilize unlabeled structured data to explicitly model the relationship between structured data and texts. To alleviate the problem of low-quality pseudo-labeled data during self-training, we further propose a novel method called Curriculum-Based Self-Training to rearrange the order of unlabeled data incorporated into self-training based on the difficulty metric. Experimental results show that CBST outperforms fine-tuning and task-adaptive pre-training methods, and achieves state-of-the-art performance in the few-shot setting of WebNLG and WikiBio datasets.

Acknowledgments

This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604) and the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096). This work was supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2019GQG1 and 2020GQG0005. This work was also supported by OPPO Research Fund. This work was sponsored by Tsinghua-Toyota Joint Research Fund.