Improving Few-Shot Text-to-SQL with Meta Self-Training via Column Specificity

1 Introduction

Text-to-SQL is a popular semantic parsing task, which aims to translate natural language questions (NLQ) into SQL programs to realize intelligent access to databases. As the most fundamental setup, single-table text-to-SQL has provided the foundation of many real-world applications, such as financial markets [Sun et al., 2020] and electric energy [Chen et al., 2021]. In this task, which is illustrated in Figure 1, all the target SQL programs follow a unified skeleton (golden box), thereby SQL generation can be decomposed into multiple sub-tasks for predicting components. Benefiting from tabular pre-trained models [Yin et al., 2020; Herzig et al., 2020; Yu et al., 2021] and encoder-subtask frameworks [Hwang et al., 2019; Lyu et al., 2020], existing methods achieve exciting results on the open-domain benchmark [Zhong et al., 2017].

However, most of them require large-scale SQL annotations as the supervision signals for training. In fact, obtaining such NLQ: How many banking companies have headquarters in the USA?

1 Code available at https://github.com/ygxw0909/MST-SQL

∗ Contact Author

rich labeled training data is not only costly, but also infeasible in many sensitive user applications due to data access and privacy constraints. In real-world scenarios, the training-used NLQ-SQL pairs for each table are often insufficient, which consequently results in over-fitting. Some works [Chang et al., 2020; Chen et al., 2021] call it a few-shot problem and demonstrate that it is the main bottleneck of single-table textto-SQL technology from theory to application.

In a few previous attempts, [Chang et al., 2020] deals with the problem by using an auxiliary mapping task while still suffering from the high cost of manually labeled annotations; [Chen et al., 2021] leverages a promising meta-learning (ML) algorithm to improve the fast adaption capability of the model. Unfortunately, the algorithm applies a one-size-fits-all updating step for all columns while neglecting the difference of their contributions, thus improving little to the task.

Self-training [Scudder, 1965] to exploit unannotated utterances provides a solution to the few-shot problem in many fields [Li et al., 2019; Qi et al., 2020; Wei et al., 2021]. For each table, unlabeled NLQs can improve model generalizability and ease over-fitting caused by limited labeled training data. During the online iteration of a text-to-SQL system, these NLQs are readily available from previous user records. In addition, we hypothesize that distinct columns contribute differently to optimization. For example, the column "Market Value" almost merely appears in financial related tables while "Date" is more common for most tables. Intuitively, to learn the generic knowledge, the object should focus on the common columns, which are closer to the center of the sample space.

Motivated by these, this paper proposes a new method, Meta Self-Training text-to-SQL (MST-SQL), to solve the few-shot problem. Initially, to separately count the contribution of

each column to the optimization, we select Hydranet [Lyu et al., 2020], a strong baseline on WikiSQL with column-wise inputs, as our basic model. Thereafter, we propose a meta selftraining framework to optimize the basic model by utilizing only a few labeled data and unlabeled NLQs. Concretely, during each epoch of training, the model first predicts the SQL programs for each unlabeled NLQ as their pseudo labels, and then updates its parameters with the loss of both the labeled data and sampled pseudo labeled data. To further improve the generalizability of the model, during the updating, we adopt a fine-grained meta-learning algorithm to guide the optimization direction. On the one hand, the algorithm is merely applied to optimize the objects directly relevant to column prediction because they are the fundamental of the whole task. On the other hand, it weighs each column by its specificity, to make the common columns contribute more greatly to the total loss. The contributions of this paper can be summarized as:

• We propose a new text-to-SQL method that performs self-training with readily available unlabeled data. To the best of our knowledge, it is the first time to leverage self-training to solve few-shot text-to-SQL.
• We propose a column specificity meta-learning algorithm for optimization, which makes the model focus on common columns to learn the generic knowledge.
• We conduct comprehensive experiments on the opendomain benchmark WikiSQL and domain-specific benchmark ESQL [Chen et al., 2021]. Our method outperforms all the baselines on few-shot tests and also maintains competitive results in the rich-resource scenario.

2 Preliminaries

Given an NLQ q and a table T = {(h1 , C 1 ), ..., (hm, Cm)}, where m is the number of the columns, hi is the i-th column header, and C i is the set of cells under hi , the goal of singletable text-to-SQL is to return a SQL query y. Each y follows a unified skeleton in Figure 1, where the tokens with $ indicate slots to be filled and * represents zero or more times.

Following previous work [Hwang et al., 2019; Lyu et al., 2020], we decompose the task into six sub-tasks: 1) Select-Column (SC): Predicting $SCOL, the column in the SELECT clause. 2) Select-Aggregation (SA): Predicting $AGG, the aggregation function in the SELECT clause. 3) Where-Number (WN): Predicting N, the number of conditions in the WHERE clause. 4) Where-Column (WC): Predicting $WCOLi , the column in the i-th condition. 5) Where-Operator (WO): Predicting $OPi , the operator of i-th condition. 6) Where-Value (WV): Extract $VALUEi from q, which is the value of i-th condition. In all the subtasks, SC and WC are most fundamental because they are directly related to table schema and other sub-tasks need to be solved by their results. Previous studies [Lyu et al., 2020; Yin et al., 2020] have proved that SC and WC play the most important roles for SQL generation.

Our work considers the following scenario for self-training: the training data is denoted by D = (A, U). Here, A = {a1 , a2 , ..., a|A| } is the set of labeled data, where ai = (qi , T i , yi ) has a gold SQL label yi . U = {u1 , u2 , ..., u|U| } is the set of unlabeled data, where ui = (qi , T i ).

3 Method

3.1 Overview

Figure 2 illustrates the overview of our proposed MST-SQL, which wraps a basic text-to-SQL model Mθ with a meta selftraining framework F. Following previous work [Hwang et al., 2019; Lyu et al., 2020], Mθ adopts an encoder-subtask architecture to generate SQL, y = Mθ(q, T ), where θ denotes the model parameters. To break its bottleneck in few-shot scenarios, F optimizes θ by learning from only a few labeled data A and additional unlabeled data U,

where θ∗ denotes the parameters with a strong generalizability.

3.2 Basic Model

We adopt Hydranet [Lyu et al., 2020] as Mθ because of its column-wise input form. Specifically, an original input (q, T ) is decomposed into m column-input (q, hi ). As shown in Figure 2(a), Mθ consists of an encoding module and a subtask module.

Encoding Module

This module is a pre-trained RoBERTa [Liu et al., 2019], used to obtain the hidden states of each (q, hi ). In view of the improvement brought by using table contents [Chen et al., 2021], we construct the input sequence of tokens, including an symbol [CLS], a column type ti , a column header hi , table contents, and the NLQ q. Here, the contents are k cells in C i with the highest literal scores [Chen et al., 2021].

RoBERTa converts each input token into a hidden vector by attention mechanism. The obtained hkq ∈ Rd denotes the vector of xkq , the k-th token in q, and hi[CLS] ∈ Rd is regarded as the semantic vector of the entire column-input (q, hi ).

Sub-Task Module

This module solves the sub-tasks using the hidden vectors of all column-inputs. First, considering the fundamental SC task, all the columns are ranked by

where Wsc ∈ Rd is a trainable parameter matrix. Then, hi with the highest Psc is taken as $SCOL. Likewise, Pwc(hi ∈ {$WCOLj }|q) for WC is calculated with the other parameter matrix Wwc ∈ Rd , and top-N hi with the highest Pwc are returned as {$WCOLj }. Here, the number of WHERE conditions, N, is predicted by

where Wwn ∈ Rˆn×d is an affine transformation, and ωi is the weight calculated by softmax(Wω · hi[CLS] ). Then, the remaining sub-tasks can be solved with $SCOL and {$WCOLj }. Concretely, $OPj is the operator ok ∈ O with the highest conditional probability, which is calculated by

Algorithm 1 Meta Self-Training Framework

Require: Labeled set A, unlabeled set U, validation set V, basic model M, hyper-parameters γ, σ.

• 1: InitializeMθ with random parameters θ, Acc∗ ← 0
• 2: Warm boot, trainMθ with mini-batches on A, obtain θm
• 3: while not done do
• 4: Predict pseudo labels ˆ yj ← Mθm (qj , T j ), where (qj , T j ) ∈ U, calculate confidence ψj for each ˆ yj , make up the pseudo labeled set P
• 5: A′ ← {(qj , T j , ˆ yj , 1)}, where each (qj , T j , ˆ yj ) ∈ A
• 6: Meta training on Dm sampled fromA′ ∪P, θm ← Fm(Dm, Mθm )
• 7: Db = A′ ∪P ′ , where P ′ is sampled fromP with σ
• 8: for batch B iterated from Db do
• 9: ∇θm Lb ← ∇θm ∑i ψiLb(Mθm (qi , T i ), yi )
• Evaluate B = {(qi , T i , yi , ψi )},
• 10: Update parameters with gradient descent: θm ← θm - γ∇θm Lb
• 11: end for
• 12: Evaluate validation accuracy AccV on V withMθm
• 13: if AccV > Acc∗ then
• 14: θ∗ ← θm, Acc∗ ← AccV
• 15: end if
• 16: end while
• 17: return Mθ∗

a Column Specificity Meta-Learning (CSML) algorithm Fm to improve the most fundamental SC and WC, in order achieve the fast adaption to new tables. The training data Dm consists of n ML-tasks sampled from A′ ∪ P, which are detailed in Section 3.4. Note that only Wsc, and Wwc, and the parameters of the encoding module are updated at this step, to guide the optimization direction of the entire θm. (b) Thereafter, all the parameters of θm are updated by optimizing the total loss of all the sub-tasks with the mini-batch training (Line 7-11). Here, each batch B is obtained from Db = A′ ∪ P ′ , where P ′ is randomly sampled from P with ratio σ to prevent the excessive impact of pseudo-labels. Thus, θm follows the macro direction while maintaining a stable updating.

After multiple epochs, θm with the best performance AccV is regarded as the optimal parameter θ∗ (Line 12-15).

where Wwo ∈ R|O|×d . Likewise, $AGGj is predicted by a similar way. Afterward, xkq is selected as the start of $VALj for WV, which has the highest probability

where W st wv ∈ Rd , hkq ∈ Rd is the hidden vector of xkq . The end index of $VALj is obtained in the same way. Finally, the SQL program is obtained by filling all the slots of the skeleton.

3.3 Meta Self-Training Framework

Algorithm 1 details our meta self-training framework F, used to find the optimal parameters θ∗ by mining the potential knowledge of unlabeled data in the following two stages.

Warm Boot

The goal of the warm boot (Line 2) is to obtain the stable parameters of Mθ, so as to subsequently provide the first batch of high-quality pseudo-labels. Specifically, randomly initialized Mθ is trained with a conventional mini-batch strategy on labeled data A. When the validation performance AccV reaches the defined threshold λ, stable θm is obtained.

Iterative Adaptation

At each epoch, Mθm first predicts a SQL program ˆ yi as a pseudo label for each unlabeled sample ui ∈ U (Line 4). To mitigate error propagation from noisy pseudo labels, we design ψi to evaluate the confidence of pi = (qi , T i , ˆ yi ).

where Psc($SCOL|qi ) and Pwc($WCOLj |qi ) are the ranking scores of predicted $SCOL and $WCOLj , respectively. Here, ψi merely involves SC and WC because their performances reflect the quality of the entire SQL in most cases. Finally, all the pseudo-labeled samples (qi , T i , ˆ yi , ψi ) make up a pseudolabeled set P, and the original A is also wrapped to A′ , where the confidence of each sample is set to 1.

Subsequently, A′ ∪ P is leveraged for back-propagation to update θm. Considering the priorities and characteristics of the sub-tasks, we develop a two-step updating here: (a) First, since column selection is typically table-sensitive, we propose

3.4 Column SpecificityMeta-Learning

Our CSML is performed on column-samples to focus on the independent contribution of each column. Specifically, each original sample (q, T , y, ψ) is broken into column-samples (q, hi , y, ψ) and all of them are shuffled and sampled to form n ML-tasks Dm. The task follows a common N-way K-shot setting, i.e., N tables are involved and each table provides K related column-samples.

For each (q, hi , y, ψ), unlike existing ML algorithms [Chen et al., 2021; Wang et al., 2021a], which optimize the total loss of all the sub-tasks, our CSML defines the following two binary-classification objects to improve SC and WC:

• Predict whether column hi exists in the SELECT clause.
• Predict whether column hi exists in the WHERE clause.

These two objects have a concise form suitable for using ML and the potential to improve the generalization ability of the model because of their table-sensitivity. In this way, the loss of each (q, hi , C i ) is defined as

where H(x, y) denotes the cross-entropy. ysc and ywc are the gold binary labels of the two objects, respectively, which can be easily obtained from SQL label y. Psc(hi |q) and Pwc(hi |q) are the predicted probability of SC and WC, respectively.

In order to promote Mθm to focus on generic knowledge, our CSML hypothesizes that common columns contribute more to the learning object, which is significantly different from the one-size-fits-all updating used in [Chen et al., 2021]. Concretely, for each hi , we define a column specificity

where N i is the frequency of hi, Ntotal is the total frequency, and Ndistinct denotes the number of distinct hi. The equation reflects that the ratio of the frequency of hi to the average frequency. Theoretically, the greater µi indicates that hi is more common.

Our CSML is detailed in Algorithm 2. Optimizing support set loss LS (Line 3) first provides the possible direction of θm and then optimizing query set loss LQ (Line 5) corrects the direction for generalization. Notice that the loss Li of each column-sample in CSML is re-weighted with specificity µi , thus common columns can lead optimization in a more general direction. In addition, the confidence ψ of each column-sample is also taken as a weight to reduce noise.

4 Experiment

4.1 Experiment Setup

Dataset and Evaluation Metrics

We evaluated our MST-SQL on the two benchmarks: 1) WikiSQL [Zhong et al., 2017] is currently the largest open-domain single-table text-to-SQL dataset, which contains more than 20k tables collected from Wikipedia and 56,355 / 8,421 / 15,878 NLQ-SQL pairs for training / development / test. 2) ESQL [Chen et al., 2021] is a Chinese domain-specific singletable text-to-SQL dataset containing 17 tables and 10,000 / 1,000 / 2,000 NLQ-SQL pairs for training / development / test.

We adopt Logical Form (LF) accuracy and Execution (EX) accuracy as the metrics, where the former evaluates the exact matching results between the predicted SQL and the gold one, and the latter compares the query results of the predicted SQL with the gold one.

Implementation Details

Our method ran on Tesla V100 Super GPUs. We used RoBERTa as the encoder by default. The hyper-parameters were set as follow: (1) The number of table content was set to 5. (2) The learning rate α, β, γ were set to 3e-5, 1e-5, 3e-5. (3) The weight of the loss calculation in CSML η was set to 0.5. (4) The threshold for confidence scores ζ was set to 0.1. (5) N was set to 4, and K was set to 15 / 5 for support / query set. (6) ML-task number n, sampling ratio σ, and warm-boot threshold λ were set to 80, 0.2, 80 in standard supervised setting, and the setting in few-shot tests will be discussed later.

Methods for Comparison and Ablation Settings

We compared with notable work that has reported results on WikiSQL, including X-SQL [He et al., 2019], IE-SQL [Ma et al., 2020], and SDSQL [Hui et al., 2021], SeaD [Xuan et al., 2021], SQLOVA [Hwang et al., 2019], HydraNet [Lyu et al., 2020], BRIDGE [Lin et al., 2020], TABERT [Yin et al., 2020], GRAPPA [Yu et al., 2021], TAPAS [Herzig et al., 2020], and MC-SQL [Chen et al., 2021].

To evaluate the contribution of components of MST-SQL, we add two ablation settings in the subsequent experiments: 1) Basic: trainMθ with the standard supervised learning on A instead of meta self-training. 2) ST-only: CSML step (Line 6 in Algorithm 1) is removed during the iterative adaption.

4.2 Results in Standard Text-to-SQL Setting

We first conducted the experiments in the standard text-to-SQL setting with sufficient training data. We used the full training set of WikiSQL as the labeled data A, and toke WikiTableQuestions [Pasupat and Liang, 2015] as the unlabeled data U, which provides more than 2k tables and 20k NLQs without SQL labels. Here we did not evaluate ESQL because it includes a non-negligible zero-shot challenge. The experimental results are shown in Table 1, where all the results with execution guiding (EG) [Wang et al., 2018] are discarded because EG requires that the query results of SQL are always nonempty, which is unreasonable in realistic scenarios. Although the full MST-SQL ranks third on testing in terms of both LF and EX, the ST-only setting surprisingly achieves state-of-theart results. It proves that self-training with unlabeled data can also improve the model in rich-resource scenarios. A possible reason for the drop brought by CSML is that mini-batch can optimize parameters more stably than meta-learning with sufficient training data. In addition, our MST-SQL outperforms the tabular pre-trained models (middle of Table 1) without requiring a massive text-SQL parallel corpus.

4.3 Few-Shot Tests

Compared with the standard setting, we are more concerned with the performance of text-to-SQL in real few-shot scenarios. Therefore, we reconstructed WikiSQL and ESQL to enhance the few-shot challenge. Concretely, since the original set of WikiSQL has large amounts of tables with identical schema, we merged them into approximately 9,000 new tables and randomly selected 500 of them for few-shot tests. For ESQL, we removed 2 tables of insufficient samples and retained 15 tables. For each dataset, we built four training sets by the shot number(WikiSQL: {1, 2, 3, 4}, ESQL:{5, 10, 15, 20}), and construct validation and test sets with the same setting(4 / 100 samples each table for WikiSQL / ESQL). Different from leveraging extra data (e.g., WikiTableQuestions) in the standard supervised setting, we built U with all the remaining NLQs of the original WikiSQL while masking their SQL annotations, in order to simulate user data collected from the same-origin tables in real-world applications.

The results are shown in Table 2. BRIDGE, TABERT, and GRAPPA have no results on ESQL due to no solutions to Chinese corpus. Our MST-SQL outperforms all the compared methods with significant improvement on all shot numbers. Notably, even the state-of-the-art BRIDGE still can not solve the few-shot problem well, especially when the shot number is extremely small. Since tabular pre-trained models have learned a wealth of prior knowledge, they perform better. Compared to GRAPPA which pre-trained by 866.5k examples, our MST-SQL achieves better performance while utilizing less extra unlabeled data. The bottom of Table 2 indicates that both self-training and CSML contribute to the entire model. The smaller the number of shots, the greater their contribution.

The Impact of Unlabeled Data Source

Sources of unlabeled data may influence the entire performance. To explore the impact, we compared two types of U in few-shot tests: a) Using remaining samples in the same datasets as U (same source), which is adopted in Table 2; b) Using WikiTableQuestions as U (different sources). Table 3 shows the results. Both types of unlabeled data are effective due to their extra knowledge. In comparison, the unlabeled data from the same source gains a greater contribution than that of different sources. The main reason can be that the identical distribution results in less noise.

Ablation Study of Self-Training

To further explore the conditions of self-training, we trained the model with different warm-boot thresholds λ ({45.0, 55.0, 65.0} for WikiSQL, {20.0, 30.0, 40.0} for ESQL), and discarded the sample confidence ψ. In addition, we compared the different ML-task numbers n of Dm and sampling ratios σ of Db. Note that all the experiments here were in the hardest 1-SHOT setting for WikiSQL and 5-SHOT for ESQL. The results are illustrated in Figure 3. Lowering the threshold results in a slower convergence speed and a worse final performance. The greater effect of λ on ESQL reveals that stability is more important for domain-specific scenarios. The performance drop of w/o CF indicates the necessity of the confidence scores. Moreover, blindly increasing the sampling ratio and the number of ML-tasks does not always bring improvement.

Ablation Study of CSML

We also analyzed CSML to evaluate its effectiveness. The experiments follow the SHOT setting in the previous section. Specifically, we compared three settings: a) w/o OB denotes using the unified CSML for all the sub-tasks instead of the defined objects in Section 3.4. b) w/o CS means that samples are not weighted with the column specificity µ. c) w/o OB & CS represents deleting both objects and µ, i.e., the ML algorithm proposed in [Chen et al., 2021]. Table 4 shows the performance of all the sub-tasks in different CSML settings. MST-SQL equipped with all the components achieves best on overall LF and most sub-tasks. Abandoning the new objects causes drops in almost all the sub-tasks, which reflects the importance of the adaptation to the sub-task differences. In addition, column specificity not only benefits SC and WC but also contributed to other sub-tasks. It can be due to improvements in SC and WC that raise the caps of other sub-tasks. Benefiting from the above improvements, our CSML gains better results for few-shot text-to-SQL than the existing ML algorithm [Chen et al., 2021].

5 RelatedWork

Single-table text-to-SQL task recently attract lots of attention since the release of WikiSQL [Zhong et al., 2017]. Previous work can be mainly divided into two directions. The earlier methods [Dong and Lapata, 2018; Zhong et al., 2017] use sequence-to-sequence models to generate SQL token by token, which is called the generation-based method. But the sketchbased method first proposed by SQLNet [Xu et al., 2017] shows a better performance and is followed by many later works [Hwang et al., 2019; He et al., 2019; Lyu et al., 2020]. Latest end-to-end works propose to use additional annotated features [Ma et al., 2020], auxiliary tasks [Hui et al., 2021; Xuan et al., 2021], and special decoding mechanisms [Wang et al., 2018; Lin et al., 2020] to further improve the effectiveness of standard supervised learning.

In recent years, tabular pre-trained models have been proposed and released. TABERT [Yin et al., 2020] presents a joint understanding of textual and tabular data, where table content is leveraged into representation. TAPAS [Herzig et al., 2020] is proposed for Table QA, but the table encoding ability can also be used in text-to-SQL. GRAPPA [Yu et al., 2021] combines table semantic parsing with a grammar-augmented pre-training framework.

The few-shot problem in text-to-SQL is first mentioned by [Chang et al., 2020], which proposes a mapping auxiliary task to enhanced the model while still suffers from the cost of additional annotations. MC-SQL [Chen et al., 2021] handles the problem by a coarse-grained ML algorithm. However, its improvement is not obvious because of its one-size-fitsall updating for all the columns. Similarly, the ML used in DGMAML [Wang et al., 2021a] is also coarse-grained and relies on multi-domain scenarios.

Self-training [Scudder, 1965] is promising for few-shot problems [Li et al., 2019; Wei et al., 2021; Qi et al., 2020]. Recently, some methods [Wang et al., 2021b; Hu et al., 2021] integrate meta-learning to it to further improve the generalization capability. The most significant difference between these methods and our MST-SQL is that they only treat ML as a tool to evaluate the sample confidence while we use column specificity to guide the ML process and optimize the model for fast adaption to new tables.

6 Conclusion

In this paper, we presented a new meta self-training method for the few-shot problem of single-table text-to-SQL. The proposed self-training process helps the model learn the generic knowledge from readily available unlabeled data. At each epoch, the model first predicts pseudo-labels and then performs a two-step updating to optimize the parameters. A fine-grained meta-learning algorithm is employed to re-weigh the columns by their specificity in order to make sure that the common columns have a greater contribution to the optimization objects. The experimental results on both open-domain and domain-specific benchmarks indicate that our method provides a promising way for few-shot text-to-SQL. In future work, we will try to expand the proposed method to apply to the joint queries of multiple tables, in order to handle more complex text-to-SQL tasks.

Acknowledgements

This work is supported by the NSFC (Grant No. U21A20488, 62006040), the Project for the Doctor of Entrepreneurship and Innovation in Jiangsu Province (Grant No. JSSCBS20210126), the Fundamental Research Funds for the Central Universities, and ZhiShan Young Scholar Program of Southeast University.