Keep Skills in Mind: Understanding and Implementing Skills in Commonsense Question Answering

1 Introduction

Commonsense Question Answering (CQA) aims to answer questions that require varieties of commonsense knowledge and skills [Talmor et al., 2019; Talmor et al., 2021]. Closedbook CQA, one of CQA subtasks that measures a model's understanding and question-solving ability without external retrieved information, has been gaining increasing attention from both academia and industry in recent years [Roberts et al., 2020; Petroni et al., 2019].

∗Corresponding Author.

Currently, most Closed-book CQA approaches improved performance by using knowledge generated by pre-trained models. Some methods [Liu et al., 2022a; Wang et al., 2022] used a pre-trained model to generate background knowledge relevant to the question, and utilized them as additional input for subsequent question answering. The other common methods [Jung et al., 2022; Wei et al., 2022] designed a series of explanatory prompts to mimic the question solvingprocess. However, relying too heavily on the knowledge generated by pre-trained models may ignore the crucial skill information that is vital in solving CQA questions. Skills are learned response patterns, which are also critical when solving CQA questions by humans [Moore, 2006]. For example, the process of thinking about "counterexample" in response to the keyword "always" in Figure 1 is a skill. Many previous studies [Talmor et al., 2019; Talmor et al., 2021] have pointed out that certain types of commonsense skills are necessary to arrive at the right answer. Meanwhile, other studies [Yoran et al., 2022; Puerto et al., 2023; Trivedi et al., 2022] have also shown that endowing reasoning skills to pre-trained language models (PLMs) can improve abilities to answer questions in other tasks. Therefore, skills, as crucial information, should be incorporated into the question-solving process.

As shown in Figure 1, when answering the question "Do you agree with the statement, a dog is always bigger than a cat?", we can identify the two key statements "always" and "bigger than" using our previously acquired skills. Because "always" implies without exception and "bigger than" is a comparison of size, they can recall the corresponding skills "plausibility" and "comparison". After that, with these two recognized skills, we can realize the need to focus on the counterexamples ("always") and the properties of the entities ("comparison"). As a result, we can correctly answer the question by utilizing skills and paying attention to considerations. Specifically, we pay more attention to counterexamples and consider the size properties of cats and dogs. This highlights the importance of identifying the skills required for answering commonsense questions in CQA tasks. However, to the best of our knowledge, there are few works in this area.

There are three main challenges inherent in utilizing commonsense skills in CQA tasks. First, most commonsense questions often do not have labels indicating the skills related to it, which creates a huge obstacle on how to proceed with further use of skills. Second, commonsense questions often require multiple skills [Talmor et al., 2019; Talmor et al., 2021]. For example, the question in Figure 1 involves both "plausibility" and "comparison" skills. Therefore, how to combine multiple required skills is a problem that needs to be solved. Third, after acquiring the skills required for the questions, how to use a variety of skills to guide answering the questions is another challenge.

To tackle the above challenges, we propose the Dynamic Skill-aware Commonsense Question Answering (DSCQA). First, DSCQA adopts a commonsense skill extraction module to extract features from the training set as skill representations. Then, these representations are used to determine the demand for various types of skills for the current question by calculating the similarity between skill representations and the question. Second, to address the challenge of combining multiple skills, DSCQA designs a dynamic skill module to generate dynamic skill representations. In this module, various types of skill representations are fused to form dynamic skill representations according to the current question. Third, to deal with the challenge of how to use skills to help the question-answering process, DSCQA utilizes perception and emphasis module which uses all skill representations (i.e., dynamic skill representations and various skill representations) by incorporating them into the model's encoding and decoding process. To sum up, the main contributions can be summarized as follows.

• We propose a novel approach of incorporating skills into commonsense question answering in order to increase the logic of the model's solution. To the best of our knowledge, this is the first work to explore the effect of skills in Closed-book CQA tasks.
• We present Dynamic Skill-aware Commonsense Question Answering framework in which the model can understand and implement skills in Closed-book CQA.
• Extensive experiments demonstrate that the performance of DSCQA surpasses other baselines on the CSQA2 and CSQA datasets under comparable conditions.

2 RelatedWorks

Closed-book Commonsense Question Answering. Many studies explore commonsense question answering in Closedbook condition where no additional knowledge is retrieved to help answer questions. Some studies [Petroni et al., 2019; Onoe et al., 2021] pointed out that a large amount of commonsense knowledge is stored in the parameters of largescale pre-trained language models. Therefore, many researches explored using pre-trained language models to generate knowledge relevant to the current question. For instance, GKP [Liu et al., 2022a] used the prompt method to conduct large-scale pre-trained models with a question to generate additional background knowledge. Chain-ofThought Prompting [Wei et al., 2022] generated a series of intermediate reasoning processes. ALEAP [Wang et al., 2022] used iterative selection to make better use of the knowledge generated by the model. Selftalk [Shwartz et al., 2020] presented an unsupervised method to produce question clarifications and appends them as external input. Different from previous methods, our model extracts features from the training set to get skills to help answer questions.

Skill Enhancement. As a psychological concept, skill is learned response pattern. It has been widely used in many subjects and fields. In Natural Language Processing (NLP) field, there are two main flavors to use skill to help downstream tasks. One type of research focused on having the model learn examples containing various skills to improve the model's reasoning skills. TeaBReaC [Trivedi et al., 2022] proposed to help improve the performance of the model by letting the model learn elaborately designed question-answer pairs containing multiple reasoning patterns. PReasM [Yoran et al., 2022] employed some predefined templates corresponding skills to extract information from the table and generated question-answer pairs for the corresponding skills. The other common research focused on the characteristics of specific skills, thus adding specialized modules. MetaQA [Puerto et al., 2023] utilized expert agents obtained by using multiple models trained on the respective category corpus (e.g., QA datasets, Google queries) to represent various skills. Then, answer selector chose from the answers provided by each expert agent and gave the final answer. NumNet [Ran et al., 2019] introduced a heterogeneous directed graph to improve the numerical skills of the model. Since commonsense questions are difficult to identify the required skills, our framework places greater emphasis on perceiving the corresponding commonsense skills.

3 Method

3.1 ProblemFormulation

We focus on commonsense question answering tasks in the form of multiple-choice and yes or no questions. For multiple-choice questions, given a commonsense question q and a list of candidate answers C = {c1, c2, . . . , ci}, our goal is to identify the correct answer among them. For yes or no questions, given a commonsense question q, our task is to judge yes or no. Since we study the problem in the Closedbook condition, we do not retrieve any additional knowledge or other datasets to help answer the questions.

3.2 Overview

The pipeline of our framework is shown in Figure 2. It can be viewed as three parts: (1) Commonsense Skill Extraction Module uses features extracted from the training set to generate various skill representations; (2) Dynamic Skill Module utilizes various skill representations to generate dynamic skill representations specific to the current question; (3) Perception and Emphasis Module takes advantage of dynamic skill embeddings and various skill representations to help solve questions. Next, we will describe each part in detail.

3.3 Commonsense Skill ExtractionModule

In order to identify the skills required for a question, it is necessary to understand the characteristics of each skill. However, the semantic features of questions with the same skill in various datasets vary greatly. That's because the datasets vary in question structure and focus. Therefore, the skill characteristics in different datasets need to be obtained separately to represent each skill's characteristics better. There are many studies [Chen et al., 2022a; Liu et al., 2020; Liu et al., 2023; Yue et al., 2022] that enrich the representation by extracting features from the datasets or documents. Here, in order to obtain the skill characteristics in the corresponding dataset, we first construct a skill seed dictionary based on the prior knowledge and data analysis, and then these skill seeds are used to further obtain the skill represetations.

Skill Seeds. Inspired by CSQA2 [Talmor et al., 2021], we extracted a series of common words or phrases related to skill statements from dictionaries and language learning websites. After that, by combining these words or phrases with the features of commonsense questions, we derived a set of skill seeds through filtering. As shown in Table 1, skill seeds are a set of representative words or phrases of the corresponding skill type questions. After getting skill seeds, we can use regularization-based selectors to assign some pseudo skill labels to those questions in the training set successfully identified by the regularization. That is, if a skill seed appears in the question, the label to which the skill seed belongs is temporarily assigned to the question. For questions that do not contain any of the above skill seeds, we assign unknown labels to them.

Skills Representation. We use the same type of questions to enrich the representation of the corresponding skills. First, we utilize the pseudo labels (including unknown) given by the skill seed to group the questions. Then, considering that the questions containing the same type of skills have certain similarities in the semantic space after encoding, similar to previous researches [Yu et al., 2022; Ye et al., 2022; Yang et al., 2018], we aggregate the questions in the same group so that the aggregated representation is taken as the representation of the corresponding skill. Specifically, we encode questions by existing context encoder (e.g., sentence-T5 model [Ni et al., 2022]), which takes average pooling of the output question word vectors {w1,w2, ...,wn} as the representation of the input questions. The question representation qi and skill representation svj is:

As a result, we obtain a collection of skill representations that have the characteristics of the corresponding dataset. These skill representations originate from the questions in the training set, so they have homology with the questions in the dataset. Therefore, the similarities between skills and the current question can reflect the demand degree of each skill for the current question.

3.4 Dynamic Skill Module

As we discussed in Section 1, when answering a specific question, people will consider and apply the skills corresponding to it. Previous researches [Zhang et al., 2022; Wang et al., 2023] have also shown that incorporating dynamic semantics can help downstream tasks. Therefore, we consider informing the model of the required skills corresponding to the question. To be specific, we propose dynamic skill module to obtain the dynamic skill representation corresponding to the question. It extracts features from each skill representation according to the similarities between the current question representation and each skill representation, thus obtains the dynamic skill representation which is dynamically generated according to the current question.

Specifically, consistent with the method of obtaining skill representations, we apply the same context encoder to encode questions and take average pooling of the question word vectors as question representation q. Then, the question representation q serves as the query, and each skill representation {sv1 , sv2 , . . . , svm } functions as the key and value for attention calculations. Consequently, these features are extracted from each skill representation to obtain the question specific dynamic skill representation dsvi :

3.5 Perception and Emphasis Module

In this subsection, we will introduce perception and emphasis module which uses all skill representations obtained from commonsense skills extraction module and dynamic skill module to help the model solve the commonsense questions. This module exploits commonsense skills from two perspectives, which we will introduce step by step below.

In order to make the model perceive the corresponding skill to the question during the encoding and decoding process, following [Clive et al., 2022], we map skill representations into prefixes so that the question can be inferred under the control of skills. In this way, we guide the question-solving process in the desired direction and provide the model with skill attribute-level information.

General Prefix. We use Prefix-tuning [Li and Liang, 2021] to learn task-level information instead of the usual finetuning. By doing so, we only need to fine-tune a small set of prefix parameters (general prefix), while keeping the parameters of the pre-trained language model frozen. This not only reduces computational costs but also facilitates the learning of MLPs that map dynamic skill prefixes. Specifically, we add a pair of trainable continuous prefix tokens {Pg, P ′g } for encoder and decoder. When doing attention calculations in the i-th layer, the current K, V vectors will be updated:

where K′ i , V ′ i ∈ R(L+M)×d , L is the length of the general prefix, d is the dimension of the hidden layer, M is the number of tokens associated with keys and values.

Dynamic Skill Prefix. We introduce dynamic skill prefixes to make the model aware of the skills corresponding to the current question. Specifically, we use MLPs to map dynamic skill representation dsvi to dynamic skill prefix form {Pd, P ′d }, which is concatenated to K and V in each attention layer of encoder and decoder, respectively, together with the previous general prefix:

where K′′ i , V ′′ i ∈ R(Ld+L+M)×d , and Ld is the length of the dynamic skill prefix.

Our method differs from GTEE-DYNPREF [Liu et al., 2022b] , which obtains the current context-specific dynamic prefixes from multiple task prefixes. And these multiple task prefixes need to be trained separately. While our approach just needs to train the mapping function from the dynamic skill representations to the dynamic skill prefixes, which eliminates the need for multiple training sessions. In addition, our dynamic skill prefixes introduce extra dataset features that are extracted from the training set data instead of being randomly initialized.

Skill-aware Keyword Focus. Humans can use existing similar memories to quickly focus on the core of the question and some can even affect the answer. Since skill representations mentioned earlier are obtained by extracting features from the training set, they can also be perceived as abstract memory storage for this particular skill. Therefore, by referring to the studies about information interaction [Zhang et al., 2019; Chen et al., 2022b; Zhang et al., 2021], we introduce skills to help the model focus on the key points of the question. Considering that different skills focus on different aspects, we use various skill representations to re-weight the individual words of the question which have been encoded by the encoder. Specifically, by modifying the method of BiDAF [Seo et al., 2017], we use each skill representation to do attention calculations for each word of the question to get the weight assigned by each skill to the current word:

where Sij represents the similarity between the i-th question word and the j-th skill representation. α is a function that can be learned to compute the similarity of two input vectors. After each word in the question gets the weight assigned to it by different skills, each word takes the maximum weight assigned to it as the final weight assigned to the word:

where b represents the final weight assigned to each question word, S represents the similarity matrix of question words and skill representations, maxcol denotes the function that takes the largest element in a column.

4 Experiments

In this section, we first present the profile of datasets and some Closed-book CQA methods. After that, by conducting extensive experiments on DSCQA and comparing with various baselines, we explore the following questions:

• Q1: How does DSCQA perform compared to other Closed-book CQA methods?
• Q2: How well do the various modules in DSCQA work?
• Q3: How does DSCQA specifically perform on questions across skill categories?
• Q4: What is the effect of different dynamic skill prefix lengths on the results?
• Q5: How does DSCQA correctly answer confusing questions compared to basic models?

The code is at https://github.com/BAOOOOOM/DSCQA.

4.1 Dataset

We use two widely-used commonsense datasets, i.e., CommonsenseQA (CSQA [Talmor et al., 2019]) and CommonsenseQA 2.0 (CSQA2 [Talmor et al., 2021]), as benchmarks. CommonsenseQA is a widely-used commonsense dataset that consists of 12,247 multiple-choice questions. Its questions and answers are based on related concepts in ConceptNet [Speer et al., 2017], including two other concepts connected to the concepts in question and two artificially created concepts are used as wrong options. CommonsenseQA 2.0 is a more challenging dataset for answering commonsense questions. It includes 14,343 yes or no questions that are made by people in order to make it hard for AI to get the answers right. In particular, since our research focuses on using skills to help with commonsense question answering, we use a regular matching method with certain skills to preliminarily classify questions in CSQA and CSQA2, and the statistics are demonstrated in Table 2.

4.2 ComparisonMethods

Our study focuses on commonsense question answering in the Closed-book setting. Therefore, we have selected the following methods that do not depend on retrieving external knowledge for comparison.

• Direct Inference under Fine-tuning. We use T5-large [Raffel et al., 2020] to perform inference directly by fine-tuning on the corresponding training set without introducing relevant external knowledge.
• Prefix-tuning [Li and Liang, 2021]. It is a novel prompt-based approach that keeps the parameters of language model frozen and fine-tunes a small continuous vector of prefixes during the training.
• Self-talk [Shwartz et al., 2020]. This method generates a sequence of information search questions by combining the current question with a predefined question prefix. After that, these questions are used to inquire zero-shot language models to produce additional relevant background knowledge, which is passed along with the questions to the pre-trained model for fine-tuning.
• GPT-3 [Brown et al., 2020]. This method uses some demonstrative prompts to derive fixed GPT-3 to generate relevant background knowledge. The knowledge and questions are then used together for model training and inference. Following the implementation of ALEAP [Wang et al., 2022], we use ten sampled knowledge spans as implementation questions.
• ALEAP [Wang et al., 2022]. It utilizes fixed GPT-3 to generate question-related knowledge and selects suitable knowledge to help solve the question by alternately optimizing the knowledge selector and answer predictor.

4.3 Implementation

In our experiment, we use T5-large [Raffel et al., 2020] as our backbone, which has 1024 dimensions hidden representations. Considering that the datasets have different answer forms, we treat CSQA as a generation problem and CSQA2 as a classification problem. We take the best result from the training process as the final result. We use AdamW [Loshchilov and Hutter, 2019] as the optimizer and set the learning rate to 1e-5. We set the maximum length of the model input to 64. We use fine-tuned Sentence-T5 [Ni et al., 2022] as our context encoder to get the representation of the question words and use the obtained representation to represent the sentence and skill representations later. We modify the OpenPrompt [Ding et al., 2022] and use it as a framework for a series of prefix-tuning in Section 3.5. For general prefixes, the prefix length is set to 100, and its dropout rate is set to 0.5. The number of attention heads is set to 12 for question-skill attention and 8 for skill-question attention.

In addition, Prefix-tuning [Li and Liang, 2021] mentioned that directly optimizing the prefixes would lead to unstable and degraded performance, so we followed their advice to use multilayer perceptron (MLP) to reparameterize general prefixes. To be specific, we initialize a smaller matrix ˜ P and then reparameterize the prefix matrix P = MLP( ˜ P). Once the training is complete, we will keep only the final prefix matrix P and discard the intermediate matrix ˜ P .

4.4 Main Results (Q1)

The experimental results for CSQA and CSQA2 are presented in Table 3. In general, our model outperforms other baselines under the same setting. For CSQA2 dataset, DSCQA outperforms the Prefix-tuning T5-large baseline by 1.78% and the previous strong baseline ALEAP [Wang et al., 2022] by 0.93% on the test set. For CSQA dataset, our method has 2.21%, 1.15% performance improvement compared with Prefix-tuning T5-large and ALEAP, respectively. By making the model perceive the skill corresponding to the question, so as to notice the notes or tricks of the corresponding skill, the performance of DSCQA is improved compared with other methods. In particular, compared with CSQA2, DSCQA has a greater improvement on CSQA dataset. We consider that the questions in CSQA2 are generated by humans against AI, the questions are more difficult and thus the skills in the questions are more difficult to identify. Moreover, compared with previous methods, our model no longer requires the assistance of large-scale PLMs (e.g., GPT-3), but achieves better results, which reflects that only using PLMs to generate knowledge has limited performance improvement, and the perception and use of various skills is necessary.

4.5 Ablation Study (Q2)

To assess the efficacy of the individual modules within our model, we remove the dynamic skill prefix and the question word re-weight part in perception and emphasis module. In addition, the effectiveness of the dynamic skill prefix is further investigated by experimentally modifying the skill prefixes. Specifically, we explored four additional configurations in comparison to our original approach. For the setting of removing dynamic skills, it is to investigate the effectiveness of making the model aware of the current skill. For the setting of removing question words re-weighting, it is to explore the effectiveness of skills in helping the model focus on the question focus. For the configuration that do not use skill classification, its purpose is to demonstrate that the improvement of DSCQA mainly depends on the skills, rather than the non-answer question features extracted from the training set. For the configuration which uses static skill labels instead of dynamic skill prefixes, it is to explore whether skill representations that include dataset features have more question affinity than pure text labels. The experimental results are presented in Table 4.

When we remove the dynamic prefixes, we see a 0.43%, 1.49% and 0.90% decline in CSQA2 development set, CSQA2 test set, CSQA development set, respectively. This suggests that informing the model skills for the current question and applying it to the inference process can help improve performance. After removing the module that re-weight the question words, performance shows declines of 0.27%, 1.29% and 1.07%. The results demonstrate that skills can help questions find key information.

For the setting that does not use skill classification, we treat all questions of the training set as a large skill category, and then obtain a overall skill representation through the commonsense skill extraction module. Following this, we utilize the overall skill representation to replace the individual skill representation for the experiment. In this way, the features of the training set are preserved, but no skill classification is performed. For the setting of static skill labels, we make use of each skill seed to classify the questions through regular matching. We then incorporate each pseudo label into the model together with the question. In particular, since this pseudo labels corresponding to the questions have no semantic information, we do not re-weight the question words in this setting. As shown in Table 4, the performance of the CSQA2 development set, CSQA2 test set, and CSQA development set decreases by 0.79%, 2.63% and 0.57% respectively after removing the skill classification. It indicates that only extracting features from the training set without skills is of limited help or even harmful to solving questions. In addition, the performance of using static skill labels decreases by 0.67%, 1.98% and 1.39% compare to DSCQA without using question word re-weighting, which indicates that extracting features from the training set can help skills guide question reasoning. All in all, with the above experimental analyses about DSCQA and its ablation variants, we thoroughly demonstrate the effectiveness of different modules.

4.6 Performance in Various Skill Categories (Q3)

In order to better evaluate the impact of our model on question using skills, we further compare the performance of DSCQA and Prefix-tuning in various skill categories on CSQA2 and CSQA development sets. Table 5 illustrates the accuracy results of DSCQA and Prefix-tuning on various skill class questions. We observe an overall performance increase in accuracy on questions across skill categories, which shows that our approach has a positive effect on helping to answer questions in the majority of skill categories. In particular, we observe that for questions involving the comparison category, the performance of DSCQA is degraded compared to Prefixtuning in CSQA2. We consider it may be because the comparison type questions involve comparing attributes to each other, which requires the model to have a deeper knowledge of the individual attributes. Therefore, for comparison skill questions, it is not enough to just let the model know that the question involved corresponds to the skill and highlight the keywords related to the skill. At the same time, it is observed that the accuracy of DSCQA for questions with skill category unknown is also improved by 3.60% and 1.93% on CSQA2 and CSQA, respectively. This indicates that extracting information from questions in each skill category is also helpful for solving questions where the skill category is difficult to determine. These results based on skill type are consistent with our assumption that the introduction of skills in commonsense question answering improves performance.

4.7 Hyper-parameter Analysis (Q4)

In this part, we investigate the influences of different dynamic skill prefix lengths on the performance on the CSQA2 of CSQA datasets. Figure 3 shows how the accuracy varies with the length of the dynamic skill prefix. We can see that the accuracy progressively decreases as the skill prefix length grows. We analyze that two factors may cause this. First, the length of the question itself is relatively short, and too long dynamic skill prefixes will cause the model to focus too much on the skill and ignore the question itself. Second, the process of expanding dynamic skill prefixes requires learning, which increases the difficulty of learning other modules. Therefore, we choose 1 as the length of the dynamic skill prefix in the DSCQA framework.

4.8 Case Study (Q5)

We use case studies to further demonstrate the role of skills in question-solving. As shown in Figure 4, the question "Peter is always a name of a man?" belongs to the plausibility category question, and the word "always" in the question determines the answer. Prefix-tuning method predicted answer is "Yes", while DSCQA predicted answer is "No". Clearly, our method predicted the correct answer. Since the pre-trained model has seen the male name "Peter" more often during training, it is more likely to assume that the statement is true. However, if the model does not realize that the question involves the "plausibility" skill, then it is likely to ignore the inherent requirements and not make the right choice. For question "Where is likely to not just have a kosher restaurant?", Prefix-tuning may overlook the "negation" skill corresponding to the word "not" and thus chooses "Jerusalem" which is more semantically similar to "kosher restaurant". These two examples demonstrate that after being endowed with skill information, the model is more likely to recognize the underlying requirement and thus is more likely to pick out the correct answer.

5 Conclusion

In this paper, we studied the use of skills in Closed-book CQA and proposed the DSCQA model. Specifically, we extracted features from the training set to form individual skill representations that help questions identify the required commonsense skills. We introduced dynamic skill prefixes to make the question aware of the current corresponding skill during inference. In addition, we used skills to re-weight the question words to help the model find keywords in the question. Experimental results showed that introducing skills can help commonsense reasoning. To the best of our knowledge, we are the first to attempt to use skills in Closed-book CQA tasks.

Acknowledgements

This research was partially supported by grants from the National Key Research and Development Program of China (No. 2021YFF0901003). This work was supported by Ant Group through CCF-Ant Research Fund.