Beyond Pure Text: Summarizing Financial Reports Based on Both Textual and Tabular Data

1 Introduction

The general purpose of abstractive text summarization is to generate concise summaries that well preserve both salient information and the overall semantic meanings of the given documents. Different from the extractive text summarization, which directly extracts the key sentences from the input documents, the abstractive text summarization is generated by abstracting the textual content at a semantic level.

Prior work has demonstrated that the abstractive summary generators could be trained in either supervised [See et al., 2017; Xu et al., 2021] or unsupervised [Nayeem et al., 2018; Chu and Liu, 2019; Tampe et al., 2022] manner. The merit of unsupervised abstractive summaries is that no domain experts are needed to craft the summaries to train the model, and thus could largely save the human annotation cost. To this end, a good number of pre-trained models, e.g., [Zhang et al., 2020a; Liu and Lapata, 2019; Lewis et al., 2020], have been proposed by taking the self-supervised information to guide the unsupervised model learning process. Not surprisingly, a very large corpus set is usually collected in their work to pretrain the generative language model (LM). On the contrary, the supervised models require human annotators to summarize a small portion of their data as the ground truth to supervise the later model learning process. Although the human cost is high, the quality of the generated summaries are reasonably higher than those generated in the unsupervised manner.

∗Corresponding author. Email: zhangxiaofeng@hit.edu.cn

Different from the pure text summarization task, our proposed financial report summarization task contains two unique research challenges: (1) our task is to abstract both tabular data and textual data, whereas the conventional approaches only abstract textual data; and (2) our adopted textual data is assumed to contain selective bias issue induced by the diversified domain knowledge of human writers. For the first challenge, our financial reports usually contain rich data such as charts and tables to provide the factual data. As aforementioned, most existing approaches are proposed to abstract pure textual data [Zhang et al., 2020a; Lewis et al., 2020; Liu and Lapata, 2019] and could not be extended to our task. We also noticed that there exist a few related approaches generating text according to tabular data [Moosavi et al., 2021; Chen et al., 2022]. However, none of them are delicately designed to directly abstract both tabular and textual data [Zhang et al., 2020a; Lewis et al., 2020; Liu and Lapata, 2019]. For the second challenge, the textual content of our financial report data is assumed to contain selective bias issue, as illustrated in Figure 1. To write financial reports for the same table, writer A1 may focus on factual data 1 as highlighted in red color according to his/her domain knowledge as well as the past expertise. Meanwhile, A2 and A3 might focus on the purple and orange tabular data. We model this phenomenon as selective bias issue, e.g. the human writer may only choose part of the tabular data to generate the corresponding financial report. Thus, both the original tabular data and the textual data should be utilized to generate a unbiased summary, and which motivates our work.

To the best of our knowledge, we are the first to raise this research problem, i.e., summarizing both tabular and textual data. A novel approach is proposed to generate Unbiased financial report Summaries using both Tabular and Textual data, abbreviated as USTT. Without loss of generality, our proposed "table+text → summary" (TTS) problem has three core sub tasks: (1) preserving salient information; (2) covering the overall semantic meanings; and (3) alleviating the selective bias issue. To preserve the salient information, if tabular data and textual data are coupled with each other, then such data should be considered as the salient information. For those textual data that are not highly correlated with the tabular data, we consider them as external knowledge-driven content written by the human writers based on their background domain knowledge. To preserve the overall semantic meanings, these external knowledge-driven content should be utilized to guarantee the coverage. To alleviate the selective bias issue, the original tabular data is assumed to contain complete factual data which is not fulled addressed by the given textual data. Thus, we use all historical textual and tabular data to approximately build an external unbiased knowledge base. For each queried tabular data, the retrieved sub set of documents from this base are used to form the unbiased set of documents to train the model.

The major contributions of this work are summarized as follows.

• To the best of our knowledge, we are the first to propose the "table+text→summary" task to directly summarize financial reports using both tabular and textual data.
• We proposes a novel approach to simultaneously abstract financial reports using both tabular and textual data. The proposed approach consists of several components designed to detect salient information, and the selective bias issue is partially alleviated using an external unbiased knowledge base.
• We have constructed a financial report summarization dataset and made it publicly available 1 . Extensive experiments are performed and the promising experimental results demonstrate the effectiveness of the proposed approach w.r.t. a number of automatic and human evaluation metrics.

1 https://huggingface.co/datasets/wza/USTT

2 RelatedWork

In the literature, text summarization has long been investigated which could be roughly classified into extractive [Kupiec et al., 1995; Saini et al., 2018] and abstractive [Gui et al., 2019; Gui et al., 2018; Nallapati et al., 2016; See et al., 2017] summarization techniques. A typical solution for extractive text summarization task is to extract the best matched sentences from source articles. Numerous works have been proposed for this task, such as integer linear programming [Galanis et al., 2012] and various graphbased techniques [Erkan and Radev, 2004; Litvak et al., 2010; Mihalcea and Tarau, 2004]. Alternatively, abstractive text summarization task is to generate concise text preserving the semantic meanings of the given documents [See et al., 2017; Dong et al., 2021], and thus has attracted increasing research efforts. The dominating techniques for abstractive text summarization are various sequence-to-sequence (seq2seq) based models [Fu et al., 2020; Jangra et al., 2020a; Jangra et al., 2020b; Klein et al., 2014; Krizhevsky et al., 2017]. Generally, seq2seq-based approaches employ a LSTMmodel to encode input text into a fixed length embeddings and decode the summaries from the learnt embeddings in a generative manner. The attention mechanism [Bahdanau et al., 2015] plays a key role in preserving salient information as well as guaranteeing the coverage, and thus is widely adopted in related approaches [Gui et al., 2019; Fu et al., 2020]. Recently, a number of graph-based approaches have been proposed for text summarization by treating sentences as graph nodes, and their edges denote their interaction behaviors. The hidden interactive patterns are assumed to contain both salient information as well as global semantic meanings. Beyond text summarization, recent approaches have focused on multi-modal summarization tasks, e.g., image-text to summary [Chen and Zhuge, 2018] and image-video-text to summary [Jangra et al., 2020b] problems. Although there exist various summarization techniques for different types of data, none of them are proposed to simultaneously abstract both tabular and textual data similar to our financial report summarization problem.

3 Formulating the TTS Problem

Let D = {⟨tl, dl, sl⟩}Nl=1 denote the annotated "table+text → summary", where tl and dl respectively denote a table and its corresponding text, sl denote the human-written summary containing ml words denoted as w1,w2, ...,wml . The purpose of the proposed TTS problem is to learn a model with parameters θ∗ to maximize the probability P of generating summary sl given tl and dl. Accordingly, our proposed TTS problem could be formulated as,

where w<q denotes the words before wq.

4 The Proposed Approach

The proposed approach is detailed in this section which consists of four components: (1) an encoder component to encode both tabular and textual data; (2) a salient content detection component; (3) an external knowledge base; and (4)

a novel unbiased summarization generator. The proposed architecture is plotted in Figure 2, with each component discussed in individual subsections for clarity.

4.1 Encoding Tabular and Textual Data

To encode tabular data, it is a natural choice to perform the encoding processing along the column-wise and row-wise. Our employed table encoder contains two sub encoders: row encoder and column encoder.

Row encoder and column encoder. Similar to [Yin et al., 2020], our row encoder is built as follows. First, we linearize each cell into 'row name|column name|cell value' and concatenate each cell representation with '[SEP]'. Then, the linearization results are fed into an encoder to acquire the embedding of a cell ci,j, where i and j respectively denote the row and column index. To differentiate the cell importance in a row, a row-wise attention is performed [Gong et al., 2019] with the cell score calculated as αi,j,j ′ ∝ exp(cTi,jWaci,j ′), where Wa is a trainable parameter matrix. Thus, the weighted cell representations are computed as vrow i,j = ∑j ′ ,j ′ ̸=j αi,j,j ′ci,j ′ . By applying a MLP layer, the embedding of a cell ci,j is finalized to crow i,j = tanh(Wb[ci,j; vrow i,j ]). After acquiring the embeddings of each cell, a row Rowi is embedded using a mean pooling operation over all its cells, given as

Similarly, we could build the column encoder and the corresponding column embeddings are directly given as

where Wc and Wd are trainable parameters.

Encoding textual data. We employ a BERT based encoder to encode text dl and the mean embeddings of all tokens are considered as the text embeddings e(dl).

4.2 Detecting Salient Content

To extract salient tabular or textual data, we treat a row or a column as a basic unit to represent certain semantic meanings. Similarly, each sentence in the text dl is treated as a basic textual unit. If the semantic meanings of a table unit are aligned with those of certain textual units, these aligned units represent highly correlated (coupled) content and therefore they are considered as salient content.

However, such alignment should be processed in an asymmetric way, i.e., "from table to text" and "from text to table". The perspective "from table to text" could capture which part of document is relatively important w.r.t. certain tabular data. The sentences with a higher score indicates a higher probability to become salient textual data. Accordingly, the perspective "from text to table" could capture the salient tabular data. Let matrix R ∈ Rd×O , C ∈ Rd×U and K ∈ Rd×L respectively denote the row, column and sentence embeddings. To calculate the pair-wise importance between a row/column and a sentence, we propose a salient detector gate which is computed as follows.

Salient detector gate between row and text. We first calculate the similarity matrix A ∈ RO×L measuring the distance between a row and a sentence, computed as

where We ∈ Rd×d is the weight matrix. The gate from row to text is then calculated as

where parameters Wf ,Wg ∈ RO×d and ωa ∈ RO , αr→k is the score associated with each sentence by setting a row as the anchor. Similarly, the score from text to row could be

Salient detector gate between column and text. Similar calculations could be performed to compute the salient detector gate between column embeddings and sentence embeddings, and the score αc→k (from column to text) and αk→c (from text to column) are directly given as

Filtering salient and external content. We empirically set a threshold γ to decouple tabular and textual data into salient content and external content. It will separate the salient information shared between the text and tabular data from the external knowledge-driven content. This process enables our model to focus on the most relevant information when generating summaries. The salient tabular data could then be represented as

where I() denotes a mask function setting the corresponding position to 1 if αk→r ≥ γ holds, ';' denotes concatenation operation and ⊙ denotes an element-wise multiplication.

Eq. 4 means that if the score of the gate is higher than the threshold, its embeddings will be kept as salient content. On the contrary, the tabular data with its score is lower than the threshold will be filtered as external content, formulated as

Similarly, the salient and external textual data are respectively computed as

4.3 Building Unbiased Knowledge Base

Building knowledge base. To partially alleviate the selective bias issue, an external domain knowledge base (KB) is built in this section. Generally, experienced domain experts are desired to build such domain KB which inevitably involves a high annotation cost. To save such human cost, it is naturally to assume that all the historical data, which contains documents generated by different experts to interpret similar tabular data, could be used to build such unbiased KB. This unbiased KB G = {⟨gu⟩}Uu=1 is built as follows. First, we concatenate the external table embeddings with their corresponding external text embeddings, written as

where u denotes the index of the unbiased data. Note that we dynamically update this KB during the model training. Empirically, we set its capacity to 5,000 pieces of data, and a FIFO (first in first out) strategy is adopted to update the KB.

Retrieving knowledge base. To query this KB, T ex in Eq.5 is used to probe this base. The best matched results in the KB are considered as the similar historical data and their associated text embeddings are retrieved to form a sub set H = {⟨hv⟩}Vv=1 ∈ G which is considered as an unbiased set. We adopt cosine similarity to measure the relevance between a query variable and an existing record, written as

where σ is empirically set to 0.9 to ensure the results are highly correlated.

Detecting unbiased content. The set H is assumed to contain complementary textual content w.r.t the queried tabular data. To extract the unbiased content, we first clustered this set using the conventional K-means algorithm as

where µj is the centroid of each cluster C ∈ H. For each queried table, we could find its nearest centroid µj. Then, documents belong to that cluster (with its centroid as µj) are assumed to contain documents written by different experts. This cluster of documents is assumed to be unbiased and the corresponding embeddings of document are fed into the decoder to generate summaries.

4.4 Generating Unbiased Summaries

The optimization goal is to maximize the saliency and coverage and minimize the selective bias simultaneously. To this end, the input of the employed decoder consists of three parts: salient embeddings, coverage embeddings and unbiased embeddings. The process for constructing these embeddings, as well as the design of the model loss, are detailed below.

Building salient embeddings. Both salient tabular Tsa and textual data Dsa are used to decode the summaries, and thus we concatenate these embeddings to form the salient embeddings, written as

Building coverage embeddings. Constructing the coverage embeddings is not as straightforward as building the salient embeddings. Typically, the source embeddings of both the table and text are used. However, both of them contain redundant content, especially in tabular data, which are equivalent to noisy data to some extent. To address this issue, we use the embeddings of external text Dex . As introduced in Section 4.2, Dex is a weighted vector by assigning lower weights to less important content, and higher weights to more important content, and its original contextual order keep unchanged. Thus, the coverage embeddings is set as Ecoverage = Dex .

Building unbiased embeddings. The extracted unbiased content from external KB are assumed to provide background knowledge to the similar tabular data, Therefore, the corresponding unbiased embeddings could be computed as

Model loss. These three embeddings will be concatenated together to decode the summaries. Intuitively, these embeddings should not be equally weighted, and the weighted embeddings are given as

These coefficients, i.e., α, β, γ are used to balance the contributions of the text encoder, tabular data encoder, and external KB embeddings in the final summary. Our designed summarizer is required to generate the most possible tokens as well as minimizing the loss between generated summary ˆ s and the ground truth s, and thus the overall model loss is defined as

5 Experiments

5.1 Constructing Dataset

As there is no public dataset for our proposed problem, we manually construct this financial report summarization dataset containing both tabular and textual data. Each triplet in this dataset consists of a table, a long text and a short human-generated summary recapitulating both tabular and textual data. To build such dataset, we first crawled 25,228 financial reports. Then, we automatically filter out reports without tabular data and 19,752 raw documents are left. To generate the dataset, we hire five experienced experts to annotate these financial reports and details of annotation process are described as follows.

Extracting tabular and textual data. As each financial report (pdf file) might contain more than one table and the annotators are required to read the report carefully and manually retrieve each table and its corresponding text. Note that the table structures vary dramatically for different stock brokers, the annotators are required to keep their original structures and convert these tabular data into editable format. At last, the retrieved each pair of data is numbered in sequence order and stored in the dataset. To process each financial report, one annotator usually takes fifteen minutes, and the overall working hour for this step is about 700 hours in total.

Writing summaries. Having tabular and textual data, these annotators are required to write a piece of concise summary based on both tabular and textual data. Each annotator independently writes a summary based on his own understandings. The criterion of each qualified triplet of data is given as follows.

• The long text should be highly correlated to the corresponding table, and must have at least 200 words.
• The human-written summaries should recapitulate both the tabular and the textual data.

Quality control. Since the summary writing is a creative task for the annotators, quality control is thus very important. We hire two additional annotators with more profound experiences to score the annotated data ranging from 1 to 5 based on the following two aspects:

• Coverage: does the summary contain sufficient information from both tabular and textual data?
• Salience: does the summary contain key information conveyed by both tabular and textual data?

To measure the inter-annotator agreement, we calculate the pair-wise cohen kappa where the annotators achieved 0.7057 in the coverage score and 0.7111 in the salience score. A triplet of data whose average score is greater than 3 will be kept to construct the dataset. At last, 13,897 pieces of data triplets are collected and are randomly split into training, validating and test sets consisting of 11117, 1390 and 1390 data pairs, respectively. The statistics of our constructed dataset are reported in Table 1.

5.2 Evaluation Criteria and Experiment Settings

To evaluate the summarization results, both automatic evaluation and human evaluation criteria are adopted in the experiments. For automatic evaluation criteria, we chose the widely adopted ROUGE (R-1, R-2, R-L) [Lin, 2004] and BERTScore (BS) [Zhang* et al., 2020b], where ROUGE criteria measures the recall performance on generating n-gram terms and the BERTScore measures the semantic similarity between the generated summary and the ground truth summary. To evaluate whether the generated summary is factually consistent with the input table or not, we adopt a triple-based factual consistency metric [Huang et al., 2021] (FactScore). We respectively extract triple set Γpred and Γtab from the generated summary and the input table, and then the FactScore (FS) is calculated as

where Count() calculates the size of the set. For human evaluation criterion, we randomly select 100 pieces of data from each model and hire two groups of annotators, and each group contains 3 independent annotators. Annotators of the two group have similar backgrounds to alleviate human bias. The generated results together with their input tables and texts are anonymized and sent to another group for evaluation. The average results of these two groups are reported as our final results. The annotators are asked to give score ranges from 1 to 5 based on the following aspects: Coverage and Salience.

5.3 ComparedMethods

As there are no related work for our proposed problem, we choose a number of state-of-the-art pre-trained models in the experiments to evaluate the model performance, i.e., BART [Lewis et al., 2020], Bert2Bert [Rothe et al., 2020], T5 [Raffel et al., 2020], PEGASUS [Zhang et al., 2020a]. As these compared methods were not proposed to summarize tabular data, we use the same method to pre-process the tabular data and concatenate them with the textual data as the input of each model for fair comparison.

5.4 Automatic Evaluation Results

We respectively train all models using only textual data, tabular data and both data, the corresponding results are reported in Table 2.

Similarity evaluation results. Both ROUGE-Typed criteria and BertScore measure the similarities between the generated summary and the ground truth. From this table, we have following observations. First, it is noticed that all models achieve better model performance when fed with textual data rather than tabular data. This is consistent with the expectation that text already provides sufficient information for summary generation. Second, if further fed with both table and text, the model performance of all baselines is significantly improved which verifies the necessity to generate summaries using both text and table data simultaneously. Finally, our proposed USTT achieves the best model performance as highlighted in bold when compared with all baselines. These observations verify that our designed model could well summarize both table and text data simultaneously.

Factual consistency results. We report the factual consistency results in the last column of Table 2. It is obvious that the model performance of the second group (input:table) is much better than that of the first group (input:text). If feed the model using "table+text", the model performance could be further enhanced. This strongly verifies that it could improve the quality of the generated summaries if using the original table as input. The best compared model for this criterion is the PEGASUS, and it is clear that our proposed USTT could improve the model performance by 43.7% (9.73 vs 6.77). This result demonstrates the superiority of our designed model to directly summarize table and text simultaneously.

5.5 Ablation Study

To evaluate the effectiveness of each proposed component, we respectively remove the table, the external database, the coverage embedding and the salient embedding component, denoted as 'w/o table', 'w/o db', 'w/o coverage' and 'w/o salient', respectively . The corresponding results are reported in Table 2. Notably, the model's performance drops significantly after removing the table from its input, indicating the importance of tabular data in generating summaries. If removes the salient information, the model performance also drops a lot and we can infer that the coupled part between text and table could well preserve the discriminative content to generate summaries. Similarly, we could find that both coverage and external db components are helpful for our task. Moreover, it is also clear that the FactScore of the "USTT w/o table" drops by 51.2% which strongly shows the effectiveness of our designed model. We also observe that the coverage and salient information does not seriously affect the factual consistency, and this indicates that the pure text is not sufficient to summarize the "Text+Table" data.

5.6 Human Evaluation Results

To perform human evaluation, we first randomly choose a hundred of generated summaries and distribute them to independent annotators after anonymizing any identification information for human evaluation, and the corresponding results are reported in Table 3. First, the human evaluation results of our approach are better than all baselines w.r.t. coverage and salience criterion, and our model could achieve comparably results, i.e., 2.98 vs 2.96 and 3.41 vs 3.48, with the golden results. This verifies the superiority of our proposed approach. Second, the model performance significantly drops if removes table information as shown in the "USTT w/o table". This also verifies the importance of the table information which is consistent with our previous results. Similar observations could be found for coverage, salient and external db.

5.7 Effect of the External Database

To investigate the effect of the external unbiased knowledge base, we visualize the word cloud of the queried results returned from the database in Figure 3. The bigger the word, the more important the word. We have following two observations. First, the summary contains a good number of words that are not contained in the original text, and this shows that the external knowledge base could well provide more informative information. Second, the query results in Figure 3 (c)

contains a few words that are not contained in the original text but are contained in the summary. This indicates that the constructed external knowledge base could well provide supplementary information and thus is helpful to alleviate the selective bias issue.

5.8 Qualitative Results Analysis

Ability to generate novel tokens. Figure 5 presents the experimental results on the ability of an abstractive summarizer to generate novel tokens. Higher bars indicate a greater ability to generate novel tokens. First, we observe that machine-generated summaries contain more novel tokens than the golden one. Second, our proposed model could generate more novel terms than all the baselines which indicates that our proposed model have a better generation ability. Last, from previous experiment, we found that the external knowledge base helps to alleviate the selective bias issue, and thus we remove this component is this experiment to further investigate its effect. The results from the "USTT w/o db" experiment supports the claim that the external knowledge base plays a crucial role in improving summarization performance.

A case study. We report a case in Figure 4 for readers to subjectively evaluate the quality of the generate summary. The left in this figure is the original tabular data, whereas the right part respectively reports the generated summaries by different versions of our proposed approach. First, it is well observed that the USTT well generates a comprehensive and unbiased summary. If removes the salient detection component, most generated textual data are hallucinatory (irrelevant to the tabular data). If removes the coverage component, the results become a bit better as it could generate textual content that are partially faithful to the original tabular data. However, the generated summary contains some content about "bond" which is not mentioned in the input table. If removes unbiased component, the corresponding summary only mentions content about "food" and "medical", whereas other industries are not mentioned. If removes the tabular data, similar observations to that of "-salient" could be found. The generated text contains many hallucination data. This indicates that both salient information and tabular data are crucial to generate a better summary.

6 Conclusion

Summarizing pure text has long been investigated in the literature. However, some domain-specific applications generally require to abstract both tabular and textual data. To tackle this issue, this paper proposed a novel "table+text → summary" problem and we manually constructed a novel "table+text → summary" summarization dataset. To resolve this challenging issue, we propose a novel unbiased financial report summarizer using both tabular and textual data. The proposed USTT model consists of several delicately designed modules and the model is trained to maximize the salience and coverage and minimize the unbiased loss simultaneously. Extensive experiments have been performed on the constructed dataset and the promising results demonstrate the superiority of the proposed approach over a number of SOTA baselines.

Acknowledgements

This work was supported in part by the Shenzhen Science and Technology Program under Grant No. JCYJ20200109113201726 and the National Natural Science Foundation of China under Grant No. 61872108.