SEMv3: A Fast and Robust Approach to Table Separation Line Detection

1 Introduction

Tables serve as efficient tools for organizing and presenting crucial information in documents. The structure of a table and the content within its cells collaboratively convey the information in the table. Table structure typically refers to the relationships between cells in a table, involving the spanning of rows and columns. Table structure recognition (TSR) aims to extract inherent structure of a table from unstructured tabular data (table image or scanned document). Table structure recognition plays a crucial role in document digitization and intelligent document analysis [Chi et al., 2019; Zheng et al., 2021].

The TSR methods based on deep learning can be broadly categorized into three paradigms: image-to-markup based, region based and split-and-merge based. Image-to-markup based TSR methods [Zhong et al., 2020; Smock et al., 2022] directly generate table structure tokens from table images, which exhibit weaknesses in handling large tables due to the length of generation. Region based TSR methods [Long et al., 2021; Liu et al., 2022] initially detect the cell boxes and then predict the logical adjacency of the detected table cells. However, in detecting table cells, they face challenge in robustly handling cases where visual clues are not obvious, such as wireless tables or blank cells. Split-and-merge based TSR methods [Zhang et al., 2024; Wang et al., 2023; Lyu et al., 2023] initially employs a split sub-network to detect table row and column separation lines. Subsequently, these separation lines are intersected to delineate the table grids. A merge sub-network is then utilized to merge the oversplit table grids into table cells. Recently, split-and-merge based approach for TSR has attracted a lot of research attention.

∗Corresponding author

In the pipeline of split-and-merge based solution, the results of table separation line detection highly impact the accuracy of table structure recognition. Due to the diversity in table styles, accurately detecting table separation lines is challenging. As illustrated in Fig. 1, the prevailing approaches for table separation line detection can be categorized according to the following types. The first type methods [Schreiber et al., 2017; Zhang et al., 2022; Ma et al., 2023] address the separation line detection through semantic segmentation, which require complex mask-to-line post-processing to parse table separation lines, as shown in Fig. 1(a). The second paradigm conceives the table separation line detection task within the framework of instance segmentation [Zhang et al., 2024], as shown in Fig. 1(b). Each separation line instance has a similar slender appearance. In order to distinguish different instances, position-sensitive instance representation features are important. However, position-sensitive instance information is difficult to learn during the training process based on instance segmentation loss. This leads to less robust separation line detection results compared to our method. The third paradigm formulates separation line detection as a line regression problem [Lin et al., 2022].

In order to strengthen the robustness of separation line detection, we introduce a new Keypoints Offset Regression module (KOR) in this paper. This module approaches separation line detection from a line regression perspective, as shown in Fig. 1(c). Specifically, KOR represents a separation line as keypoints and regresses the offsets between the line and its keypoint proposals to accurately locate the line. Compared to other segmentation-based separation line detection methods, our approach eliminates the need for complex mask-to-line post-processing and is more robust. In contrast to previous line regression methods [Lin et al., 2022; Wang et al., 2023; Lyu et al., 2023], we do not directly predict the absolute coordinates of line keypoints within the table image. Instead, we predict the offsets from the proposals, effectively incorporating the location prior of the keypoints, which reduces the difficulty of regression.

Following the split stage, the over-split table grids are merged in the merge stage. In order to achieve a precise and effective merging of these table grids, we introduce the concept of a "merge action" to denote the merging operation for each table grid. We utilize only simple convolutional layers in the construction of the merge module to predict the merging actions of the grids.

We amalgamate the Keypoints Offset Regression module (KOR) with the merge module, constituting an end-to-end table structure recognition system denominated SEMv3. Our primary contributions are delineated as follows:

• We propose Keypoints Offset Regression module (KOR) to robustly and quickly detect the table separation line. KOR represents separation lines with a series of keypoints, locating these keypoints by regressing the offset from proposals to keypoints.
• We define the merging action of each grid to describe table structure, which can effectively improve the accuracy of merging.
• Our proposed SEMv3 attains state-of-the-art results on prominent public benchmarks, including ICDAR-2019 cTDaR Historical [Gao et al., 2019], WTW [Long et al., 2021], and iFLYTAB [Zhang et al., 2024]. SEMv3 excels in diverse and challenging TSR scenarios, encompassing wireless tables, and deformed tables.

2 RelatedWork

2.1 Image-to-markup Based TSR

Image-to-markup based TSR methods [Zhong et al., 2020; Nassar et al., 2022; Huang et al., 2023] use encoder-decoder architecture to generate table structure markups from a table image. However, auto-regressive decoding the structure markups causes error accumulation and inefficient inference. DRCC [Shen et al., 2023] adopts a cascaded two-step decoder architecture, predicting row tags first and then cell tags for each row in a semi-autoregressive manner. This approach alleviates the error accumulation problem specific to autoregressive models. OTSL [Lysak et al., 2023] introduces a new table structure representation language, reducing the difficulty of decoding structure tokens. Nevertheless, Image-tomarkup based TSR methods require a substantial amount of training data to achieve optimal performance.

2.2 Region Based TSR

Region based TSR methods [Xue et al., 2019; Zou and Ma, 2020; Prasad et al., 2020; Xiao et al., 2022] initially identify primitive regions, such as text segments or cell boxes, and subsequently recover the logical relationships among these primitive regions. To achieve accurate primitive region detection, methods like [Raja et al., 2020; Qiao et al., 2021; Zheng et al., 2021] leverage the characteristics of cell arrangement to introduce additional constraints. However, these constraints are effective only for tables without deformations. For a more precise recovery of the logical positions of primitive regions, FLAG [Liu et al., 2021] and NCGM [Liu et al., 2022] enhance the feature representation of nodes in the relational graph. LORE [Xing et al., 2023] employs a cascaded structure to regress the logical position of cells. Limited by cell detection, the performance of these methods in handling blank cells and wireless tables still needs improvement.

2.3 Split-and-merge Based TSR

Split-and-merge based TSR methods detect table separation lines to parse table grids, then merge the over-split grids into table cells. In the split stage, early methods [Schreiber et al., 2017; Tensmeyer et al., 2019; Zhang et al., 2022] assume that the table lines are aligned, so they could not handle tables with deformations. In order to address deformed table samples, RobusTabNet [Ma et al., 2023] utilized spatial CNN (SCNN) [Pan et al., 2018] to enhance the feature representation of separation lines. TRACE [Baek et al., 2023] predict the mask of visible and non-visible lines separately. But RobusTabNet and TRACE require complex post-processing of mask-to-line. SEMv2 [Zhang et al., 2024] addresses separation line detection in the way of instance segmentation. SEMv2 aggregates instance convolution kernels with global information and then obtains separation line instancelevel masks by convolving instance convolution kernels with instance-independent feature maps. However, the positioninsensitivity of the instance convolution kernels results in less robust line detection. TSRFormer [Lin et al., 2022] is the first to define table separation line detection as line regression and uses Dert to directly regress the position of keypoints. TSRFormer with DQ-DETR [Wang et al., 2023] introduces dynamic object queries to gradually regress keypoints. Nevertheless, this approach reduces the efficiency of line regression. GridFormer [Lyu et al., 2023] completes splitting by directly predicting the coordinates of grid corners.

In the merge stage, RobusTabNet and TSRFormer predict the merging relationship between grids using a relational network [Zhang et al., 2017], which do not take advantage of global features of grid cells structure. SEMv2 predicts a merging map for each grid, which is redundant. RNN networks [Cho et al., 2014] are used in SEM [Zhang et al., 2022]

to predict cell merging map serially. It is time-consuming and needs additional text information from Off-the-shelf OCR engine.

3 Method

As illustrated in Fig. 2, SEMv3 extracts visual features from the backbone. The split module predicts the row and column separation lines by regressing the offsets between proposals and the keypoints. And then the separation lines are intersected to obtain the table grids. The merge module merges the over-split grids into cells by predicting the merge actions based on starting grid of grids.

To obtain the visual features of the input table image, we use Resnet-34 [He et al., 2016] with FPN [Lin et al., 2017a] as backbone to generate four different levels of features P2, P3, P4, P5. Subsequently, we amalgamate the information from all levels of the FPN pyramid into a single output F ∈ R H2 × W2 ×C . H and W represent the width and height of the input table image, respectively. C is the number of feature channel.

3.1 KOR Based Split Module

Within the split module, two parallel branches are attached to the shared feature map F for the detection of row and column separation lines, respectively. Each branch includes two modules: a feature enhancement module to enrich the context information of F, and a keypoints offset regression module (KOR) aimed at detecting the separation lines. In the following sections, we will elucidate the details of these two modules by using the row separation line detection branch as an illustrative example.

Feature Enhancement Module

Inspired by RobusTabNet [Ma et al., 2023], We employ the Spatial CNN module [Pan et al., 2018] to enhance the feature F. The spatial CNN module enhances visual features through efficient row and column messaging. We combine repeat 4 down-sample blocks [Lin et al., 2022] and two cascaded spatial CNN into feature enhancement module. We obtain the row start point detection feature F row sd ∈ R H2 × W32 ×C and the row line regression feature F row lr ∈ R H2 × W32 ×C through two row feature enhancement modules with non-shared parameters based on row information passing. Likewise, we calculate enhanced features F col sd ∈ R H 32 × W2 ×C and F col lr ∈ R H 32 × W2 ×C for columns.

Keypoints Offset Regression Module

Separation line representation. To unify the definition of separation lines in wired tables and wireless tables, we consider the visible cell border line as the separation line in wired tables. For wireless table, we define median line of separation region [Schreiber et al., 2017] as row/column separation line. We represent the ith row separation line with keypoints {krow ij |j = 0, ...,N row k -1}. N row k denotes the number of keypoints. N row k is determined by the width W of input image and the sampling step size t, as defined in Eq.1.

The x-coordinates of the keypoints krow ij are denoted as xij, where xij = j × t. Given that the x-coordinates are fixed, the position of a row separation line can be defined solely by the y-coordinates y of its keypoints. To predict these ycoordinates, we regress the y-axis offset δrow ij between the keypoints and their proposals. The first keypoint krow i0 in the ith row separation line is designated as the starting point. The keypoint proposals share the same x-coordinates as the keypoints and have y-coordinates identical to the starting point, as shown in the Row Keypoints Offset Regression subplot of Fig. 2.

Keypoint proposal head. The x-axis coordinate of the starting point is fixed at 0. Drawing inspiration from SEMv2 [Zhang et al., 2024] and TSRFormer [Lin et al., 2022], we classify whether each row has a starting point to determine the y-axis coordinate of the starting point of the row separation line. We predict the probability P row of each row having a starting point using row-wise average pooling and a softmax operation on the row start point detection feature F row sd . Subsequently, we apply Non-Maximum Suppression (NMS) [Zhang et al., 2024] to eliminate duplicate starting point predictions. We then sample Nk keypoint proposals for each row separation line, which are horizontally aligned with the starting point.

Keypoint offset head. The deformation of table can result in that keypoint proposals do not accurately represent the curved separation line. To address this challenge, we further predict the offset between the proposals and the ground truth keypoints. The offset helps in refining the location of the keypoints. We sample features of proposals K′ ∈ RC×N row×N row k from the row line regression feature F row lr based on the coordinates of the proposals. Here, N row indicates the number of row separation lines. We obtain the whole separation line representation features S row ∈ RC×N row for each row separation line by averaging the proposals features belonging to the same row, as following:

To ensure that each keypoint can consider the information of whole line, we concatenate the line representation features of each row to the feature of the proposals in the channel dimension. In Eq.3, K denotes the keypoint feature.

Instead of using a fully connected network to predict the offsets of keypoints, we employ a 1 × 3 convolution layer on the keypoint feature K to predict the offset δij in the row separation line detection branch. This operation allows each keypoint to pay attention to the information of adjacent keypoints, thereby ensuring a smooth separation line.

After predicting row and column separation lines, we create the intersection of the row and column separation lines to obtain quadrilateral boxes of grids B ∈ RM×N×8 . M and N denote the number of rows and columns of the grid, respectively.

3.2 Merge Module

Grids Feature Embedding

The representation feature E′ ∈ RM×N×Cg of the grids is the sum of visual features extracted by the RoiAlign [He et al., 2017] algorithm and the absolute position embedding (PE) [Xu et al., 2020] of the grids. Cg is the number of the representation feature channel.

The PE employs a fully connected network to map normalized top-left corner and bottom-right corner coordinates of

(a) Table Grids

Figure 3: Definition of merge action based on grids. (a) The girds based table structure. The starting grids are painted orange. The black dashed lines represent the borders within the grid that need to be merged. (b) The merge action map.

the grid to Cg dimensions. To enhance E′ , we introduce a row/column-based self-attention mechanism, which automatically aggregates features from other grids, resulting in the enhanced grid representation feature E.

Merge Action Prediction

Inspired by OTSL [Lysak et al., 2023] and Formerge [Nguyen et al., 2023], we define the merging action of a grid to depict the table structure, as shown in Fig. 3. The upper-left grid in a cell serves as the starting grid, as illustrated in Fig 3(a). As shown in Fig. 3(b), the merging action for starting grids is Stay, implying that these grids do not need to merge either to the left or upwards. The merge action for grids in the same row as the starting grids is Left, suggesting that these grids need to merge with the ones on their left. The merge action for grids in the same column as the starting grids is Upward, indicating that these grids need to merge with the ones above them. The merge action for the remaining grids in the same cell is X, which denotes merging both upwards and to the left. This novel definition of merge action allows us to succinctly and comprehensively characterize the grid based table structure with only one merge action map, as shown in Fig. 3(b).

According to the definition of merge action, the starting grids play a pivotal role in determining the merge action of the remaining grids within the same cell. To enhance the accuracy of merge action prediction, we introduce an auxiliary branch for the classification of starting grids. This helps in identifying the starting grids during the prediction of merge action. In this branch, we transform E into the starting grid classification feature, Es, using a convolutional layer. Es is used to classify the starting grids. Finally, we concatenate Es and E channel-wise to predict the merge action.

3.3 Training Object

Starting points detection loss. We supervise the prediction of starting points using binary cross-entropy loss LBCE. This is represented as:

Although the starting point has a width of only 1 pixel, we enlarge the range of starting points for ease of training. Here, ˆP row i represents the ground truth distribution of starting points for the ith row separation line. In wireless tables, ˆP row i is set to 1 if the ith row is located in the separation region. For wired tables, ˆP row i is 1 if the ith row is located in the neighborhood of starting points with a width of 8. We also calculate the loss Lcol sp for column starting points.

Keypoints offset regression loss. ˆ δrow ij denotes to the ground truth offset between the j th keypoint proposal in the ith row separation line. The loss of row line keypoints offset is calculated with L2 loss, as:

We define the column line keypoints offset loss Lcol δ in the same manner.

Merge action classification loss. We use focal loss in in both predicting starting grids and merge actions.

The loss for starting grid classification and merge action prediction are denoted as Lsg and Lma respectively. ˆP sg is set to 1 if a grid is a starting grid. The symbol a represents a merge action, which can take one of four values: S, L, U, or X. The termLfocal refers to the focal loss [Lin et al., 2017b].

Overall loss. All the modules in SEMv3 are trained endto-end. The overall loss function is a summation of several component losses, as shown below:

4 Experiment

4.1 Implementation

We use Resnet-34 pretrained on ImageNet [Krizhevsky et al., 2012] with FPN as backbone. And the feature F channel number C is 256, grid feature channel number Cg is 512. The sampling step size t is 32. Models are trained end-to-end for 100 epochs. We use Adam [Kingma and Ba, 2014] as the optimizer. The initial learning rate is 1 × 10-4 , and the learning rate is adjusted to 1 × 10-6 according to the cosine annealing strategy [Loshchilov and Hutter, 2016]. All experiments are implemented in Pytorch v1.7.1 and conducted on 4 Nvidia Tesla V100 GPUs with 24GB RAM memory. During the training process, the ground truth grid boxes coordinates are used when extracting grid represent feature using RoIAlign.

4.2 Datasets andMetric

We evaluate the performance of our method on several public datasets. These datasets encompass a wide range of challenging scenarios related to table structure recognition.

ICDAR-2019 cTDaR Historical [Gao et al., 2019] dataset contains 600 training samples and 150 testing samples from archival historical documents. This dataset presents several challenges, including a limited number of training sets, unclear row and column separation lines, and deformation in the table area. We use the cell adjacency relationship (IoU=0.6) [G¨obel et al., 2012; Gao et al., 2019] as the evaluation metric of this dataset.

WTW [Long et al., 2021] contains 14581 wired table images collected from real business scenarios. There are seven difficult cases in this dataset: inclined tables, curved tables, occluded or blurred tables, tables with extreme aspect ratios, overlaid tables, multi-colored tables, and irregular tables. Metric on WTW is the cell adjacency relationship (IoU=0.6) [G¨obel et al., 2012; Gao et al., 2019].

iFLYTAB [Zhang et al., 2024] contains 12,103 training samples and 5,188 testing samples. The dataset includes a variety of complex examples from both electronic documents and natural scenes. Based on the table data collection scenarios and table characteristics, iFLYTAB is further divided into four subsets: Wired-Digital (WDD), Wired-CameraCapture (WDC), Wireless-Digital (WLD), and WirelessCamera-Capture (WLC). We use the cell adjacency relationship [G¨obel et al., 2012] followed SEMv2 [Zhang et al., 2024] as the evaluation metric. To evaluate the performance of the split stage, we use grid detection with an IoU of 0.9 [Long et al., 2021] as the metric. F1-G represents the F1-score for grid detection, in Table 5.

SciTSR [Chi et al., 2019] and PubTabNet [Zhong et al., 2020] are from scientific literature PDF, where table samples have aligned axis. The test subset of SciTSR, SciTSR-comp, is composed of 716 complex tables containing span-cells. We evaluate our method with the cell adjacency relationship metric [G¨obel et al., 2012] on SciTSR-comp. TEDS and TEDSStruct [Zhong et al., 2020] is used to evaluate performance on PubTabNet.

4.3 Comparison with State-of-the-arts

Our method focuses on robustly recovering the table structure from deformed and wireless tables in challenging scenarios. On the following three challenging datasets, our method achieves SOTA. On ICDAR-2019 cTDaR Historical dataset, we achieve 88.9% F1 score, as shown in Table 1. On WTW dataset, we achieve performance with SOTA, as shown in Table 2. Notably, LORE who is the second best on these two datasets takes 0.5 for IoU when evaluating models while we take 0.6. On iFLYTAB dataset, as shown in Table 3, we improve SEMv2 by 0.9% F1 score. As shown in Fig. 4, Our approach performs well on challenging table samples, e.g., deformed tables, blurred tables, wireless tables.

Our model also demonstrates comparable performance to SOTA in simpler scenarios involving electronic documents. However, the testing dataset of SciTSR has been manually rectified in various ways, resulting in an unfair comparison on the SciTSR dataset. On the PubTabNet, the performance of our method is slightly inferior to that of LORE, which belongs to the region-based methods. In the split-andmerge based methods, we achieve performance comparable to SOTA.

4.4 Ablation Studies

To verify the effectiveness of our split and merge module, we undertake ablation experiments through several systems designed as shown in Table 5. In the split stage, the Instance Segmentation (IS) based separation line detection module from SEMv2, employs dynamic convolution to generate a segmentation mask for each separation line. We add a new convolutional layer to generate instance-independent features, following the original SEMv2 settings. In this paper, we introduce the KOR module to detect separation lines by regressing the offset between the key points and their proposals. In the merge stage, we propose a merge action based merging module(MA). Also, we implemented the ParaDec merge module in SEMv2 to predict the merge map (MP) for each grid.

The Robustness of the Split Module

To establish the robustness of our split module, we compare the performances of IS and KOR under same merge mod- T4 × � × � 92.5 37.9 93.4 59.9 96.4 64.6 96.3 63.2 94.4

Figure 5: Qualitative split results of different split modules. (a) is the prediction of table grids from KOR, (b) is the prediction of table grids of table grids from IS. The red dashed box indicates lowquality results.

ules. T1 and T2 serve to compare the performance of IS and KOR under the MP merge module. On the other hand, T3 and T4 illustrate the performance of IS and KOR under the MA merge module. Under both merge module settings, our KOR consistently achieves superior split results, as evidenced by the F1-G. This is particularly notable with the wireless tables (WLD and WLC), where KOR performs almost twice as well as IS. As depicted in Fig. 5, the split results of T1 and T2 on the wireless tables clearly demonstrate that KOR produces split lines of higher quality. Furthermore, our KOR proves to be more stable compared to the IS split results when the merge module changes.

The Speed of the Split Module

The IS split module generates a mask for each table separation line, leading to a computational cost that increases linearly with the number of rows and columns. Conversely, our method consistently represents the position of lines with a single offset map, meaning the computational cost does not significantly increase with the growth in rows and columns. We evaluated the inference time of both split modules on the SciTSR test dataset, with images resized to 512 × 512. As illustrated in Fig. 6, the x-axis represents the sum of the number of rows and columns with a step of 10, while the y-axis represents the inference time. The inference time of our KOR is less affected by the increase in the number of rows and columns. Particularly in cases with a large number of rows and columns, KOR is significantly faster than IS.

The Effectiveness of the Merge Module

As shown in Table 5, MA always outperforms MP when the split module is kept constant. MA have a positive gain on the detection accuracy of separation lines with IS. Our merge module can significantly reduce the memory occupancy. The space complexity of MA is O(NM), but MP has a space complexity of O(N 2M 2 ). Benefiting from our modeling of the merge action, we can merge over-split grids with high accuracy using a much simpler network structure than SEMv2.

5 Conclusion

In this paper, we introduce SEMv3, a fast and robust table structure recognizer that adheres to the split-and-merge paradigm. In the split stage, we propose a split module (KOR) based on keypoints offset regression for robust and fast detection of split lines. KOR detects separation lines by regressing the offset between the keypoint proposals and the line. During the merge phase, we define a grid-based merge action to effectively characterize the table structure. Our method achieves state-of-the-art performance on the ICDAR2019 cTDaR Historical, WTW, iFLYTAB datasets, demonstrating its effectiveness on table samples in challenging scenarios, such as wireless tables and deformed tables. Ablation experiments show that our split module KOR is fast, robust, and achieves significant improvement on wireless tables. Our merge module is also efficient.

Contribution Statement

Chunxia Qin: Conceptualization, Methodology, Writing - original draft,review and editing. Zhenrong Zhang: Conceptualization, Methodology, Writing -review and editing. Pengfei Hu: Writing -review and editing. Chenyu Liu: Methodology, Writing -review and editing, Resources. Jiefeng Ma: Writing -review and editing. Jun Du: Methodology, Writing - review and editing.