Stage-wise Stylistic Headline Generation: Style Generation and Summarized Content Insertion

1 Introduction

Although neural-based sequence-to-sequence (seq2seq) models are able to generate factual, concise, and fluent headlines [Chopra et al., 2016], the headlines generated are austere and lack stylistic attributes to draw audiences into an article. Stylistic Headline Generation, or SHG for short, is a deep learning task proposed by Jin et al. [2020] that seeks to generate headlines with a specified style, such as humorous, romantic or clickbait. The theory of the model is very sound: headlines containing the same content as the plain headlines are generated with the desired stylistic attributes. Unfortunately, in practice, acquiring enough parallel data to train the model is almost impossible. Hence approaches to SHG that do not rely on large scale labeled training sets need to be explored [Sudhakar et al., 2019; Dai et al., 2019; Jin et al., 2020; Krishna et al., 2020].

SHG in an unsupervised setting, however, presents several challenges. The first is how to decompose stylistic patterns and summarized content during the generation process. The second is how to explicitly interpret the style generated with controlled style constraints. The third is how to ensure the content of the headline is consistent regardless of the style. Several researches have already attempted to answer some of these questions. For instance, latent space representations can be disentangled into semantic content and stylistic attributes [Zhao et al., 2018; Shen et al., 2017], but the disentanglement process can mean some of the semantic information get lost with long text strings, plus it is no longer possible to control the quality of the headlines being generated [Dai et al., 2019]. By contrast, style-specific decoder approaches can generate output according to the target style while preserving the original content to a relatively high degree [Li et al., 2020b; Jin et al., 2020]. Delete-Retrieve-Generate (DRG) [Li et al., 2018] and its extension Generative Style Transformer (GST) [Sudhakar et al., 2019] alternatively decompose style factors by wordlevel editing actions, which allows for fine-grained control over style attributes.

∗ Corresponding Author

Our approach does not directly build on any of these methods. Instead, we are inspired by the observation that certain styles of headlines share the same inherent linguistic patterns: the stem and syntax of the sentence is the same while some key words change to express the content. Figure 1 shows an example. Here, there are two article-headline example pairs. Although each of the articles is written from a different perspective, the headlines for each are written to evoke the informative style. Both of them follow the syntax 'How to [verb] [noun] __ [noun]', where the verbs and nouns represent the substantive content, i.e., find and home in Example #1, train and college course in Example #2. To simplify the syntax even more, verbs are doing words and could be replaced with the word 'do'. From this observation, we conjecture that the SHG task could be creatively framed as a stage-wise process: constructing the style first and then populating that style with substantive content. Hence, we propose an end-to-end stagewise model for the SHG task consisting of a style generation component and a content insertion component. For brevity, we name this S-SHG, short for stage-wise SHG.

The style generation component first identifies inherent linguistic patterns and generates style attributes as an intermediate output with placeholders for the content to be inserted in the second component. Generating the intermediate outputs is done through supervised signals. As Figure 1 shows, distinguishing style from content strongly relates to the parts of speech (POS) in the target sentence. Hence, we design a masking strategy based on POS tagging to transform the target sentence into its intermediate supervised signal.

To implement these two component, we leverage two individual transformer decoders-a stylistic decoder and an insertion decoder. These decoders, along with a shared encoder, are then optimized via a seq2seq headline generation task and a style-specific sentence reconstruction task within a multi-task learning framework. The approach follows an end-to-end asynchronous training strategy.

This end-to-end stage-wise approach addresses all three of the challenges associated with unsupervised data raised earlier. That is, the model is able to decompose stylistic patterns and summarized content, making the outputs interpretable. Moreover, it assures accurate style control while preserving the summarized content as much as possible. The superiority of the framework lies in its ability to explicitly learn stylistic attributes, with the supervision of the constructed intermediate supervised signals.

We evaluate the approach using the same three SHG tasks described in Jin et al. [2020]. Each task is assessed in terms of style control, content preservation, attraction, diversity, and fluency. Both automatic and human evaluations show that our model outperforms the state-of-the-art methods by all indicators. Further, a manual analysis of the intermediate outputs demonstrates interpretability and shows that the model performs well at both style generation and content insertion.

The contributions of this paper can be summarized as follows: 1) We propose a novel model called S-SHG, which consists of a style generation component and a content insertion component. 2) We inject intermediate supervised signals into the model, ensuring interpretability and the ability to decompose the headline into its stylistic attributes and substantive contents. 3) Experiments on three SHG tasks show that each component in our approach performs well on its sub-task and clarifies interpretability.

2 RelatedWork

Text Style Transfer The aim of this field is to change the style attributes of a piece of text while preserving its content. However, like SHG, it also suffers from a lack of training data that parallels the target style. Additionally, the models offer little explainability, making it difficult to separate style from content. To address the first concern, Shen et al. [2017] train a cross-aligned auto-encoder, with shared content but a separate style distribution. Hu et al.[2017] apply an attribute classifier and a VAE framework to guide the generator to produce sentences with a desired style. Similarly, Zhao et al. [2018] and John et al. [2019] use a regularized auto-encoder within an adversarial training framework. However, these methods require extensive hyper-parameter tuning and are not particular good at preserving the content. Denoised auto-encoding, another approach for generating style information, works through a self-learning-based reconstruction process [Shen et al., 2020].

Many approaches based on adversarial training have been proposed to solve the second problem, which is to find a disentangled latent space irrelevant of the style [Shen et al., 2017; Zhao et al., 2018]. However, it is difficult to assure the quality of disentanglement with these strategies. They also lack interpretability. Several groups used cycled reinforcement learning [Xu et al., 2018] to strengthen style attributes and fuse stylistic information through a style-specific decoder [Li et al., 2020a; Li et al., 2020b; Jin et al., 2020]. Others subsequently added a mapping function to modify the latent representations [Mai et al., 2020; Shang et al., 2019]. The unique point of our approach is that it avoids the limitation of phrase boundaries and offers interpretable results at our stage-wise process.

3 Our Method

This section begins with a brief introduction to SHG task. We then introduce the overall structure of our proposed S-SHG. Finally, we outline the data construction methods, the training scheme, and the inference process of the proposed model.

3.1 ProblemFormulation

The dataset for SHG includes two parts [Jin et al., 2020]. One is a parallel headline generation dataset X = {x(i), y(i)}Ni=1 consisting of pairs of news article x and their plain headlines y which have no specific style. The other is a non-parallel dataset T = {t(i)}Li=1 consisting of sentences t that do carry stylistic information in the desired style s (e.g., clickbait). Overall, the goal of the task is to generate a stylistic headline ys with style s given a source article x, where the generation can be formulated as Ps(ys|x).

3.2 Model Details

A challenging goal with our model is to explicitly show which part of sentence represents the "style" and which part summarizes the original content as the plain headline. Motivated by the phenomenon illustrated in Figure 1, we design a stage-wise process that essentially generates a stylistic pattern and then fills in the blanks with meaningful content.

Figure 2 provides a simple schematic of the process. The model comprises a shared encoder, a stylistic decoder and an insertion decoder, all of which are based on the Transformer architecture [Vaswani et al., 2017]. The source article x are encoded into z as the inputs to the decoders. The intermediate output yinter (for y) or yinter s (for ys) are generated first by the Stylistic Decoder. These intermediate output contains the stylistic attributes and masked content-relevant placeholders. Next, the Insertion Decoder fills in the summarized content words, to produce either a plain headline y or our desired stylistic headline ys.

Shared Encoder The encoder is denoted as Enc(.). It can be shared since it plays a similar role to map input article x to latent representation z when generating both plain headline y and stylistic headline ys. The mapping process can be represented as z = Enc(x).

Stylistic Decoder To better decompose the content and the style of text, our stylistic decoders generate yinter and yinter s . Following Jin et al. [2020], the decoder's parameters are shared between Decsty and Decsty s except for the normalization layer and the attention mechanism. The two sets of parameters are denoted as {α, αs} and {W q ,W q s }. Thus, the intermediate outputs yinter and yinter s are acquired as follows:

where Decsty (.) and Decsty s (.) denote the stylistic decoders for generating the plain headline y and the stylistic headline ys, respectively. Hence, our model is capable of generating observable intermediate outputs yinter s and yinter that are explainable.

Insertion Decoder The insertion decoder fills in the contentrelevant placeholders with the correct words, which shares a similar scheme with Conditional Mask Language Model (CMLM). This task needs the bi-directional information around the masked words, which is not appropriate for a common auto-regressive Transformer decoder, as it only attend to the information before the target word. Hence, we apply a non-autoregressive Transformer decoder to predict the masked words simultaneously based on bi-directional information in yinter and yinter s . Meanwhile, it can also drastically reduce the inference time.

Because similar functionality of the insertion decoder is needed to generate both the plain headline y and the desired stylistic headline ys, we use one shared insertion decoder for both tasks. The outputs are y = Decins (z, yinter ) and ys = Decins (z, yinter s ), where Decins (.) denotes the shared insertion decoder.

In summary, the stylistic decoder learns to generate the intermediate outputs containing the style attribute and contentrelevant placeholders. The insertion decoder receives the intermediate outputs and inserts words that appropriately preserve the original content into the placeholders.

3.3 Data Preparation

To have the model generate the intermediate outputs yinter and yinter s , we need supervision signals for the stylistic decoder. These signals, denoted as m, must contain both stylistic attributes and content-relevant placeholders, which can be acquired by masking certain content-relevant words from y(i) and t(i). To do this, the training corpus X is modified to ˆX = {x(i),my(i), y(i)}Ni=1 that is agnostic to styles and T is modified into a non-parallel dataset ˆ T = {t(i),mt(i)}Li=1 of a specific style. my represent the masked plain headlines of y, while mt are the masked sentences of t.

The main objective is to create intermediate supervised signals (i.e., my and mt) in both the headline generation dataset and non-parallel stylistic dataset. The core idea is to keep the subset of words reflecting stylistic attribute of a sentence and mask the other content-relevant words.

Extracting Style-AgnosticContent It is difficult to pinpoint the signals for a specific style. However, in a way, we can say that the content is the words that convey key information about meaning, irrespective of style. For the most part, these elements will be nouns, conjunctions, pronouns, and orientation phrases. For this task, we enlist the help of Stanford CoreNLP 1 to analyze the POS in the target sentences. We then mask these words with the placeholders [mask]noun, [mask]conj, [mask]pron, [mask]orient. Note that a uniformmasking probability of 0.2 is applied empirically, which means that not all the content-relevant words are necessarily need to be masked. This is to make it easier for the stylistic decoder to acquire the basic ability to generate language. This masking strategy enables the model to not only generate more fluent sequences that preserve content but also to extract the stylistic attributes from an unsupervised stylistic corpus with the first component.

In Figure 3, mt and my show an example of the intermediate supervised signals. The model first generates intermediate output given the supervised signals my and mt, e.g., "[mask]noun in northeast [mask]noun kills at least 29". This phrase includes both style-relevant structures and placeholders indicating that content words, such as words "Floods" and "China", should be placed in these positions.

3.4 Model Training

As Figure 3 shows, the end-to-end training process of S-SHG, is conducted on our two constructed datasets ˆX and ˆ T based on the multi-task framework proposed in Jin et al. [2020], where a Denoised Auto-Encoding (DAE) task and a seq2seq task are conducted simultaneously to capture style information and summarize, respectively.

We take a further step to bring our S-SHG into this training process. In this way, the learning of these two tasks are further decomposed by the stylistic generation component and the content insertion component.

1 https://stanfordnlp.github.io/CoreNLP/

Incorporating the intermediate supervised signals my and mt, we modify the original objectives P (y|x) and P (t|˜ t) into the following form:

where Eq. (2) shows the objective for the supervised headline generation task (i.e., Tasky), and Eq. (3) is for the unsupervised DAE training (i.e. Taskt). The term ˜ t is converted by injecting noise into the sentence t. Psty and Pins represent the distributions from the stylistic and insertion decoder respectively.

Given this decomposition, S-SHG captures stylistic information with the first component, then learns summarized content with the second component. Details are shown as follows:

Component - I: Style Generation

For the training of the style generation component (w.r.t. stylistic decoder), we have two intermediate signals my and mt as supervision, minimizing the cross entropy loss as follows:

Component - II: Content Insertion

For the training of the content insertion component (w.r.t., the insertion decoder), we use the original target y and t as supervision. As the task of this component is equal to a conditional mask language model (CMLM) task, we optimize it with cross entropy loss over the masked tokens in my and mt:

The whole model is trained by jointly minimizing the style generation component training loss Lsty and the content insertion component training loss Lins via the proposed stage-wise framework. The total loss becomes:

where λ is a trade-off parameter, which is set as 0.5 in experiments.

3.5 Inference

Given a news article x, the stylistic headline ys is generated by the following inference process:

Meanwhile, due to the intrinsic structure of our S-SHG model , we can also acquire a plain headline without any specific styles:

4 Experiments and Results

Experimental Settings We use the datasets published by Jin et al. [2020], which consist of a parallel dataset CNN-NYT, and three non-parallel stylistic corpora related to clickbait, humor, and romance.

The CNN-NYT dataset comprises article-headline pairs from the New York Times (NYT) and the CNN datasets. These data are cleaned by filtering out sentences of less than four tokens, leaving 134,443 training samples, 3,000 validation samples, and 3,000 test samples. Without article-headline pairs, the three stylistic corpora are made up of style-specific sentences. Specifically, the humorous and romantic corpora are collected from the novels in the corresponding genres of BookCorpus [Zhu et al., 2015]. The clickbait corpus is assembled from the Examiner SpamClickBait News dataset 2 . Each consists of 500, 000 sentences.

Baselines We compare S-SHG against the following four well-known baselines:

MASS-Stylistic [Jin et al., 2020], initialized with MASS model [Song et al., 2019] and then sequentially fine-tuned on the CNN-NYT dataset via seq2seq training followed by DAE training on the stylistic dataset.

Generative Style Transformer (GST) [Sudhakar et al., 2019], which replaces the style words of the source sentence with the other style words retrieved from target sentences. Because GST lacks a headline generation module, we fine-tuned MASS on the CNN-NYT dataset and use GST to translate its generated plain headlines into the target style.

2 https://www.kaggle.com/therohk/examine-the-examiner

Style Transformer (ST) [Dai et al., 2019], a text style transformer that incorporates style information through a reconstruction task. The stylistic headline generation is accomplished by using the same way as GST.

TitleStylist [Jin et al., 2020], the seminal model, trained on supervised headline generation and unsupervised DAE.

Note that all the models are trained on the processed datasets mentioned above.

4.1 Quantitative Results

Style Control Style transfer accuracy is often used to evaluate how well an approach controls style. The idea is to apply a high-performance style classifier to evaluate the probability that a sentence adheres to a particular style. The accuracy metric (ACC) is therefore defined as the percentage of the generated sentences assigned to be the target style by the classifier. To evaluate the accuracy, we use two types of classifiers based on BERT [Dai et al., 2019] and fastText [Joulin et al., 2017]. Both classifiers are trained on the combined plain headline/stylistic training sets, and both achieve good performance on the test sets of all the stylistic datasets 3 . The two ACC columns in Table 1 show the accuracy for all the models. Using the fastText classifier, S-SHG surpasses the rest of the baselines, by between 10.0% and 36.2%. Similar trends are reflected by ACC-BERT metric.

Analysis on Global Diversity Although S-SHG returned better ACC metrics, we want to explore the possibility that this model was simply generating repeat, unique templates. Hence, to measure whether the model is learning diverse stylistic attributes, we implement a diversity metric. Diversity is an important criterion in text generation. Here, we define global diversity as the overall diversity of the model outputs, measured by automatic calculation following the Distinct-N metric [Li et al., 2016] across the entire set of outputs. The results are reported in Table 1. S-SHG has a competitive Distinct-N score, which means that it is not generating repeat text templates. Additionally, we also define local diversity by the dynamism and richness of the content of a headline sentence. More details on this are provided in Section 4.2.

3 BERT-based classifier achieves 99.6%, 99.5%, and 96.1% while fastText attains 99.6%, 94.2%, and 99.8%, respectively.

Content Preservation We use the standard metrics BLEU [Papineni et al., 2002] and ROUGE [Lin, 2004] to measure how well the models expressed the given content. Higher scores mean better preservation. The leftmost block of Table 1 shows the results for all models. Unfortunately, we fail to reproduce the good results for TitleStylist reported in the original paper [Jin et al., 2020], using their open-source code 4 . We report our results as they stand. The results demonstrate that S-SHG preserves more content information than the other baseline models while generating both style-specific headlines and factual headlines.

Language Perplexity We use perplexity (PPL) to measure the fluency of the generated sentences. A lower perplexity indicates that generated sentences are more fluent. We finetune OpenAI-GPT [Radford et al., 2018] with our training headlines 5 to calculate the PPL of the generated headlines. The results in Table 1 show that S-SHG is comparable with TitleStylist and surpasses the other baseline models by a significant margin.

4.2 Human Evaluation

To more comprehensively assess our model, we also conduct human evaluations in addition to quantitative calculations. To this end, we randomly sample 20 news abstracts from each test set, and ask ten judges to rate the outputs generated by each of MASS-Stylistic, GST, ST, and TitleStylist, plus our S-SHG. The judges make their assessments against the following four criteria: 1) Relevance-how semantically relevant the headline is to the news article. 2) Attractiveness-how appealing they feel the headline is. 3) Fluency-how comprehensive and easy-to-read the headline is. 4) Diversity -using the principles of the local diversity, discussed in Section 4.1. Concretely, a diverse headline is a single non-trivial sentence at the lexical level [Zhu et al., 2018]. Following Liu and Lapata [2019], we use the Best-Worst Scaling method [Kiritchenko and Mohammad, 2017], asking the judges to choose the best examples of each criteria from the entire set of outputs (randomly mixed so judges do not know which model produce which headline). The final score of each model is calculated as the percentage of the times it is chosen as best minus the percentage of times it is chosen as the worst. Hence, final scores, shown in Table 3, range from -1 (worst) to 1 (best).

4 https://github.com/jind11/TitleStylist

5 PPL on the development set is 34.049.

The results of this evaluation mainly conform to the quantitative results, from which we can conclude that S-SHG generate the most reflective, appealing, fluent, and diverse headlines. Interestingly, theMASS-Stylistic model receives slightlymore votes for relevance for the Romance and Clickbait datasets than the proposed S-SHG. However, we can clearly see that sentences generated by MASS-Stylistic are too redundant to be a headline, through the examples included in Table 2. Apart from this, our method still has the second best Relevance score. This and the automatic evaluation score metric prove the effectiveness of our model.

4.3 Qualitative Results and Findings

To further investigate the style factors and performance of S-SHG, we sample several headlines generated by the models from the "clickbait" dataset, as shown in Table 2, and inspect the results manually. We draw several valuable conclusions that the community might benefit from.

Interpretable Generation As mentioned, the outputs generated by MASS-Stylistic are too long to be regarded as headlines and too incomprehensible to be of any use. The results of the GST are deemed the least fluent while the ST repeats its outputs. By contrast, S-SHG generates semantically appropriate, largely grammatically correct headlines that are understandable and appealing. Further, S-SHG produces intermediate outputs, which can be examined to reveal the elements of the headline relevant to style vs content. Take the rightmost column in Table 2 as example, 'Whats next for' are the stylistic attributes, while the content placeholders '[mask]conj' and '[mask]noun' have been generated properly and inserted as 'after' and 'Bahamas'. This makes the process of SHG interpretable and understandable.

5 Conclusion

The paper presents an end-to-end stage-wise SHG model containing the style generation component and the content insertion component, where the former generates stylistic-relevant intermediate outputs and the latter inserts the summarized content. Experimental results on three stylistic datasets have shown that our stage-wise model is fully interpretable and capable of generating high-quality stylistic headlines.

Acknowledgements

We appreciate the helpful discussions with Boxing Chen. We also thank all the anonymous reviewers for their insightful suggestions.