Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner

1 Introduction

Tables are ubiquitous for organizing and communicating structured data across diverse domains, ranging from scientific literature and business reports to web pages and financial documents [Ye et al., 2024; Zheng et al., 2021]. They store a wealth of information essential for applications such as knowledge discovery, decision support, and data-driven analytics [Shwartz-Ziv and Armon, 2022; Wang et al., 2024a]. In the context of intelligent table applications, one fundamental yet challenging task is table recognition: converting imagebased table representations into structured data formats. Over the years, substantial efforts [Salaheldin Kasem et al., 2024] have been made to address this problem, introducing various approaches to address challenges, such as image segmentation techniques and cell object detection methods.

∗Corresponding author

Recently, the advent of Large Language Models (LLMs) [Chang et al., 2024] and Vision Large Language Models (VLLMs) [Yin et al., 2024] has revolutionized natural language processing and computer vision. For LLMs, their powerful understanding and reasoning capabilities have facilitated numerous tabular data mining tasks, such as table-to-text generation [Guo et al., 2024], table question answering [Wang et al., 2024b; Mao et al., 2024], and table semantic understanding [Deng et al., 2022; Cheng et al., 2025a]. Meanwhile, several VLLM-based methods have emerged to bypass traditional OCR pipelines for visual table analysis and understanding [Hu et al., 2024].

Despite these advancements, our investigation reveals a noticeable gap: the application of VLLMs to table recognition remains underexplored. This task serves as a foundational building block for table-related applications. Some existing work [Luo et al., 2024; Zhang et al., 2024] has focused on pre-training or fine-tuning VLLMs to accomplish this task. However, fine-tuning VLLMs for specific tasks is often computationally expensive and risks catastrophic forgetting of general capabilities. To address this, we explore a generative approach that does not require additional fine-tuning, specifically leveraging a training-free paradigm using pre-trained VLLMs for table recognition. Recognizing the absence of dedicated benchmarks in this domain, we propose an evaluation benchmark based on a hierarchical design philosophy [Sui et al., 2024; Cheng et al., 2025b; Liu et al., 2024b], comprising recognition tasks for table recognition. Through extensive evaluations, we identify a critical bottleneck: lowquality input images significantly hinder the table recognition capabilities of the evaluated VLLMs.

To overcome this limitation, we propose the NeighborGuided Toolchain Reasoner (NGTR) framework for effective table recognition. One of the key features of the framework is its integration of lightweight models and the strategy of retrieval-augmented generation to improve image quality and guide structured data recognition. Specifically, we propose a preprocessing toolkit with various lightweight models to enhance input image quality. For each input instance, we retrieve a similar neighbor from the training data and use the experience gained from that neighbor to guide the generation of tool invocation plans. Furthermore, we incorporate One Scenario, One Model Various Scenarios a reflection-driven tool selection module at each step to iteratively refine the table recognition output. This enables VLLMs to produce more accurate structured data.

To validate the effectiveness of the proposed NGTR framework, we conduct extensive experiments on multiple public table recognition datasets. The key observations are as follows: (1) Our NGTR framework significantly enhances the table recognition performance of naive VLLM-based approaches; (2) While VLLMs achieve competitive accuracy on specific datasets compared to traditional models, a noticeable performance gap remains in favor of traditional models. Nonetheless, we have preliminarily revealed the performance boundaries of VLLMs in several representative table recognition datasets. As is shown in Figure 1, the VLLM-based table recognition approach demonstrates the capability for universal modeling. This method facilitates a paradigm shift in design objectives from a model-centric to a data-centric focus, presenting significant potential for further exploration. We hope this work will inspire more research efforts in the future. In summary, the contributions of this paper are as follows:

• We conduct a systematic investigation into VLLM-based table recognition by introducing a hierarchical benchmark for evaluating their recognition capabilities.
• We propose the NGTR framework to address critical bottlenecks in table recognition, such as low-quality input images.
• We conduct extensive experiments to report the promising performance and potential of VLLMs for table recognition, along with interesting observations that highlight areas for future research.

2 RelatedWork

Table Recognition. Earlier table recognition (TR) methods predominantly rely on heuristic rules [Kieninger and Dengel, 1999; Shigarov et al., 2016], these approaches rely heavily on handcrafted features or implicit rules. In the era of deep learning, numerous studies have made impressive progress in handling more intricate and heterogeneous table structures. Topdown methods [Schreiber et al., 2017; Siddiqui et al., 2019; Ma et al., 2023; Qin et al., 2024] predict table borders to infer the structure information. Bottom-up methods [Zheng et al., 2021; Qiao et al., 2021; Xing et al., 2023] first identify table cells with object detection models [Ren et al., 2016; Carion et al., 2020], and then predict the cell relations to organize the row-column structures to form the overall tables. These methods follow an explicit two-stage learning paradigm with relatively strong transferability and explainability, yet the risks of ambiguous contents and boundless structures may lead to unstable and incorrect prediction results. Recently, sequence-based methods [Zhong et al., 2020; Nassar et al., 2022; Huang et al., 2023] have been widely explored to directly generate markup sequences that define structures with specific decoders. Although these approaches require a massive of training data and computing resources, they have demonstrated substantial potential to unify visualtext parsing tasks [Wan et al., 2024].

Large Language Models. In recent years, LLMs have demonstrated exceptional performance in tasks such as multitask learning [Chen et al., 2024], zero-shot learning [Kojima et al., 2022], and text generation [Li et al., 2024]. LLMs have not only broken through the limitations of traditional technologies in processing natural language text, but have also shown capabilities in reasoning [Wei et al., 2022] and planning [Guan et al., 2023; Gou et al., 2023]. Meanwhile, VLLMs combine visual and language understanding capabilities, enabling LLMs to process visual information. For multimodal understanding scenarios (e.g., scene text recognition [Wang et al., 2011], visual question answering [Antol et al., 2015]), VLLMs have been widely validated as effective [Guo et al., 2023; Ye et al., 2023; Liu et al., 2024a]. With continuous progress in text-rich scenarios, some studies [Zheng et al., 2024; Zhao et al., 2024; Chen et al., 2023] have also focused on enabling VLLMs to handle multimodal table understanding tasks.

Despite their promising success in various domains, VLLMs applied to TR remain under-evaluated and underexplored. Our study presents a comprehensive benchmark for VLLM-based TR evaluation. Subsequently, we propose a novel framework to address the bottleneck of VLLMs, thereby enhancing their capabilities in TR.

3 Preliminary and Proposed Benchmark

3.1 ProblemDefinition

We employ the generation paradigm of VLLMs to address the table recognition (TR) task, which is formulated as a format mapping problem from images to sequences. Formally, given a TR dataset D = {(I i ,Hi )}ni=1 with n samples, we predict the corresponding structured form Hi for each table image I i . Specifically, we provide the image table I i along with a prompt P as input to the VLLMs, which generates the structured data form ˆHi = VLLM(P, I i ).

3.2 Benchmark Evaluation Setup

This section proposes an evaluation benchmark for TR based on VLLMs, outlining the hierarchical recognition tasks to assess their performance and the evaluation setup.

Recognition Task Design

We design several hierarchical recognition tasks to conduct a more in-depth assessment of VLLMs' table recognition capability, including the cell-level, row-level, column-level, and global table-level. Table 1 presents the details of the hierarchical recognition tasks.

Cell-level. We evaluate the cell-level recognition capability of VLLMs within three specific recognition tasks. Merged cell detection task aims to recognize cells that span multiple rows or columns in the table. Content/index-based cell recognition tasks evaluate the structural and content recognition capabilities of VLLMs, which are crucial for assessing whether VLLMs could perform well in fine-grained table recognition.

Row/Column-level. We evaluate the row/column-level recognition capability of VLLMs within two specific recognition tasks. Row/column index-based data recognition tasks are designed to assess whether VLLMs can accurately identify row/column elements.

Table-level. We evaluate the overall table-level recognition capability of VLLMs. The visual table size detection recognition task is designed to evaluate whether VLLMs can comprehend the global structural information and accurately determine the number of rows and columns in a table.

Baselines

In this study, we evaluate the performance of six VLLMs. For open-source VLLMs, we select Phi-3.5 (Phi) and Llama-3.290B (Llama) for evaluation. For closed-source VLLMs, we evaluat GPT-4o-mini (GPT-mini), Qwen-VL-Max (Qwen), GPT-4o (GPT) and Gemini-1.5-Pro (Gemini).

3.3 Benchmark Evaluation Results

Evaluation results of hierarchical recognition tasks are presented in Figure 2. Among all the VLLMs we selected, GPT and Gemini demonstrate the strongest performance, consistently outperforming the other VLLMs. Furthermore, the open-source Llama demonstrates a significant performance gap compared to the closed-source VLLMs. We give some highlights associated with the benchmark results as follows:

Row-column Sensitivity Analysis. We found that all models show inconsistent performance between row/columnrelated tasks. To mitigate the influence of uneven distributions of rows and columns, leading to varying difficulty levels, we further refine the experimental results by selecting samples where the difference between the number of rows and columns does not exceed three. As shown in Figure 3, models perform better on column-related tasks, suggesting VLLMs tend to favor column-structured data. A likely reason is that, in many cases, columns represent diverse attributes while rows correspond to similar entities-making attributebased (column) tasks more amenable to accurate recognition.

Image Quality Analysis. We conduct in-depth experiments and analysis on the TR task. The results demonstrate that the VLLMs perform relatively well on the SciTSR dataset with higher image quality using simple prompts. However, the performance gap is considerably more significant when processing the PubTabNet dataset with lower image quality, especially compared to traditional models. This phenomenon indicates that the quality of the input image is a key bottleneck limiting the performance of VLLMs. Section 3.4 provides a more in-depth analysis of this bottleneck.

3.4 Bottleneck Analysis

We further investigate the performance of VLLMs under varying image quality conditions and assess their visual robustness to these conditions through empirical analysis.

Experimental Setup

To comprehensively evaluate the visual robustness of VLLM, we focus on three distinct visual challenges: the image quality challenge, the table border quality challenge, and the geometric deformation challenge. Details of these challenges are provided in Table 2.

Visual Conditions. Visual conditions is a key factor affecting the accuracy of VLLMs in table recognition. To assess it, we systematically analyze the performance of VLLMs under various visual conditions across three scenarios: blur, underexposure, and overexposure. These analyses demonstrate the robustness of VLLMs in handling impaired visual conditions.

Table Border Quality. Table borders indicate structural information in table elements. We evaluate the impact of border visibility and completeness on table recognition performance by considering scenarios including unclear table borders and missing borders. Additionally, we explore the effect of border changes in the table by thickened table borders.

Geometric Deformation. Geometric deformation caused by viewing angles or operations can disrupt the geometric consistency of tables. We evaluated the robustness of VLLMs against geometric deformations by testing them under tilt conditions of 20° and excessive tilt conditions of 40°.

Discussion and Analysis

As shown in Figure 4, while blurring and overexposure can degrade text clarity to some extent, the distortion caused by skewed tables severely disrupts structural integrity, significantly affecting the recognition performance of VLLM.

Although table borders are often considered essential for conveying structured information, fading or removing these borders has minimal impact on the performance of VLLMs. This result suggests that VLLMs do not heavily rely on borders. VLLMs only pay slight attention to the structural information the borders provide when thickening.

4 Neighbor-Guided Toolchain Reasoner

Through our in-depth analysis of VLLMs' performance on the benchmark, we have identified that improving input image quality is essential for enhancing VLLMs' capability to recognize and interpret structured image data more effectively. To address this, we propose the NGTR framework.

4.1 Framework Overview

NGTR enhances input image quality by applying various tool combinations tailored to low resolution, overexposure, and noise interference. As shown in Figure 5, we design a similarity-based neighbor retrieval module to select a suitable combination of tools. Subsequently, the tool invocation experience learning module executes each plan and generates the corresponding structured data to evaluate the effectiveness of different plans. Finally, we propose a reflection-driven tool selection module to integrate iterative tool invocation and dynamic feedback to refine the processing flow. The optimized image is then input into the VLLMs, which utilize their powerful reasoning capabilities to generate structured data.

4.2 Toolkit Preparation

Inspired by the conclusions of the Bottleneck Analysis (Section 3.4), we employ five distinct tools to address various scenarios and potential issues that may arise in the table recognition task. These tools are shown in Table 3. By combining different tools, NGTR effectively addresses various challenging situations. For example, when the table border is faint, the VLLMs can invoke the border enhancement tool to strengthen the structural information by thickening the table border.

4.3 Similarity-based Neighbor Retrieval

Neighbor retrieval methods enable VLLMs to retrieve similar neighbor samples, providing richer contextual information. In the NGTR framework, we hypothesize that images with similar features exhibit similar results after being processed by the same image preprocessing toolchain. Consequently, the processing results of neighbor samples could guide the selection of a potentially optimal toolchain for test samples. We first retrieve images that are similar to the target task images from a sample set. Subsequently, we employ a prompting template to leverage the VLLM's planning capabilities to generate tool invocation plans. The retrieval process can be formally described as follows:

Retrieval(I test , D′ , f ) = arg max [f (I test , I i )]|D′ | i=1 , (1) where I test represents an image from the test set, D′ denotes a subset of the training dataset, and f is the similarity measurement function. In this paper, we combine the ORB (Oriented FAST and Rotated BRIEF) algorithm with the Hamming distance as f to measure the similarity between images. Then, we guide the VLLMs to generate multiple tool invocation plans for the image. The generation process can be formally expressed as follows:

where T represents the description information set of all available image preprocessing tools, including their functions, applicable scenarios, invocation identifiers, and other relevant details; N(I test ) denotes the neighbor image samples of the test sample I test , along with their associated features, retrieved from the training set; and {p1, p2, . . . , pn} represents the generated candidate set of plans, which are then used to select an appropriate tool invocation plan.

4.4 Tool Invocation Experience Learning

In this module, we follow a sequential workflow to evaluate the multiple tool invocation plans. First, we execute each tool invocation plan generated by the previous module to obtain multiple processed images:

where fpi denotes the image preprocessing tools. Next, we employ a prompt template to guide the VLLMs in generating a markup sequence. Subsequently, we evaluate the prediction results based on the example labels. The evaluation process employs the tree edit distance-based similarity (TEDS) metrics to quantify the accuracy of the VLLMs output. By following this process, we calculate a quantitative score for each toolchain, enabling the selection of a suitable plan.

4.5 Reflection-driven Tool Selection

Although the tool invocation experience learning module provides a high-quality plan, mindlessly applying the tool invocation plan to new samples may result in the loss of critical information in the image, thereby affecting the accuracy of the final result. To address this, we introduce the reflectiondriven tool selection module during the execution phase to refine the processing flow, reduce information loss, and thereby improve recognition accuracy. The formalized expression of the reflection module is as follows:

Let I (t-1) denote the image before the t-th operation and I (t) denote the image after the t-th operation. The VLLMs computes γ (t) to determine whether to accept the operation:

where γ (t) is a binary decision indicating the quality change between the before and after images. The function reflect(·) evaluates the difference in quality. If γ (t) = 1, the operation is considered successful; otherwise, if γ (t) = 0, the operation is rejected, and the process proceeds to the next step.

This step-by-step module enhances the interaction between the VLLMs and the target image, ensuring the accuracy of the final task outcome. More importantly, introducing this module enables downstream researchers and developers to flexibly customize and expand the toolkit without worrying about the impact of poorly performing expanded tools on the final results, thereby significantly improving the versatility and transferability of our framework. In the last step, we design a simple prompt template to instruct VLLM to generate a markup sequence and obtain the result of table recognition.

5 Experiments

5.1 Experimental Setup

Datasets

In this study, we utilize three widely-used table recognition datasets: SciTSR [Chi et al., 2019], PubTabNet [Zhong et al., 2020], and WTW [Long et al., 2021], each offering unique characteristics and challenges. SciTSR is a dataset comprising tables extracted from the scientific literature, and the image quality in this dataset is relatively high. In contrast, the image resolution of PubTabNet is 72 pixels per inch, and its overall image quality is relatively low. WTW contains images collected from the wild, introducing a variety of extreme cases, such as tilt, blur, and table curvature. These datasets encompass diverse table types and various unique visual challenges, providing a robust foundation for benchmarking.

Baselines

We select six Vision Large Language Models (VLLMs), including Phi 1 , Llama 2 , GPT-mini, Qwen 3 , GPT 4 , and Gemini 5 , as baseline models for comparison. Additionally, we select three representative deep learning-based methods as baselines for comparison: EDD [Zhong et al., 2020] based on sequence modeling, LGPMA [Qiao et al., 2021] based on cell bounding box detection, and LORE [Xing et al., 2023] based on cell point center detection.

Evaluation Metrics

As for the evaluation metrics of TR, we use a similarity metric based on Tree-Edit Distance (TEDS) [Zhong et al., 2020] and the TEDS-Struct metric. We employ two evaluation metrics for the hierarchical tasks described in Section 3.2: accuracy (ACC) and micro-averaged F1 score (F1-score). Specifically, ACC is used to evaluate cell-level tasks (excluding merged cell detection) and table-level tasks; the F1-score is utilized for row- and column-level tasks and merged cell detection.

Implementation Details

For the PubTabNet dataset, We randomly select 1,500 images from the validation set. For the SciTSR and WTW datasets, we use their complete test sets for evaluation. Since the WTW dataset does not provide content information for table recognition, we do not report its TEDS scores.

For LORE, since it is mainly aimed at table structure recognition but not table content recognition, we only report its performance scores for table structure recognition. As for EDD, since its model training requires a large amount of end-toend annotated data, and SciTSR and WTW lack corresponding labeled data, its performance on these datasets has not been evaluated. LGPMA depends on table content for training, but since WTW lacks content labels, its performance on this dataset was not assessed.

1 https://azure.microsoft.com/en-us/products/phi/

2 https://www.llama.com/

3 https://qwenlm.github.io/blog/qwen-vl/

4 https://openai.com/index/hello-gpt-4o/

5 https://deepmind.google/technologies/gemini/pro/

5.2 Main Results Analysis

Tables 4 show our benchmark results on the table recognition and table structure recognition tasks. Based on the experimental results, we draw the following insights:

Performance Analysis of NGTR

As shown in Table 4, the experimental results compare our NGTR framework with baseline methods. The main results show that our framework achieves significant performance improvements on the PubTabNet dataset, mainly attributed to our framework's enhanced VLLMs robustness when dealing with low-quality inputs. On the SciTSR dataset, our framework also outperforms all VLLMs baselines, further verifying our framework's effectiveness.

Open-source and Closed-source VLLMs

The advanced open-source VLLMs demonstrate capabilities comparable to closed-source VLLMs in this task. As a representative of open-source models, Llama exhibits outstanding performance, particularly in the table structure recognition task (TEDS-Struct) on the PubTabNet dataset. Llama achieves a relatively better result, surpassing GPT by 2.73 points and Gemini by 2.36 points. These results confirm the potential of open-source VLLMs for further research.

WTW Dataset: A Challenge for VLLMs

Table 4 presents the experimental results on the WTW dataset. These results indicate that prompt-tuning based VLLMs methods still exhibit significant gaps compared to traditional lightweight OCR methods, highlighting the challenges VLLMs face when processing datasets with wild scenarios. Further analysis of the outputs suggests that performance declines notably when VLLMs process tables with numerous empty cells, unevenly distributed text, and skewed, rotated, or densely packed text. Further analysis of the outputs suggests that VLLMs tend to ignore cells lacking semantic content. While this behavior helps avoid processing irrelevant data, it also limits their ability to effectively capture the structural information of tables with many blank cells.

5.3 Ablation Study w.r.t Key Components

Effectiveness of Tool Invocation Experience Learning. We performed an ablation study by removing the tool invocation experience learning module. In this experiment, we only led the VLLMs to generate a tool invocation plan and applied it directly to the test samples. As shown in Table 5, while the VLLMs could generate a valid tool invocation plan, the lack of effective validation of the generated plan led to a significant performance decline. This further validates the critical role of the tool invocation experience learning module in improving the NGTR framework performance.

Effectiveness of Reflection-driven Tool Selection. We performed an ablation study by removing the reflection-driven tool selection module. We apply all the tools to the task images in a single pass. The results are presented in Table 5, without the reflection module for stepwise backtracking validation, VLLMs cannot effectively supervise the processing procedure, which may lead to the incorporation of unsuitable tools for the current sample, thereby affecting performance.

5.4 Hyperparameter Sensitivity Analysis

The NGTR framework contains two core parameters: the maximum length of the toolchain execution plan L and the number of plans generated each time N. As shown in Figure 6, a moderate toolchain length achieves an adequate balance between complexity and performance, as excessive toolchain length increases combinatorial complexity and limits processing performance, thereby affecting the framework's ability to generate high-quality solutions. Similarly, generating a moderate number of execution plans effectively balances solution quality and generation efficiency, whereas generating too few or too many plans slightly reduces performance. Therefore, a moderate toolchain length and number of execution plans can balance complexity and performance well, providing valuable guidance for the tool invocation.

6 Conclusion and Limitation

This paper addressed the under-explored challenge of table recognition using VLLMs in a training-free paradigm. We proposed the NGTR framework, which enhanced input image quality through lightweight models and neighbor-guided tool invocation strategies. Extensive experiments demonstrated that NGTR significantly improved VLLM-based table recognition performance. This work not only established a benchmark for table recognition but also highlighted the potential of VLLMs in advancing table understanding, paving the way for future research and applications.

Limitation. Despite the strengths of our framework, we acknowledge several limitations that warrant further investigation. Firstly, its performance depends on the underlying toolkit. Secondly, when the available set of neighbor candidates does not sufficiently cover a wide range of scenarios, selecting an inappropriate neighbor may lead to suboptimal performance. Nevertheless, we believe the NGTR framework demonstrates strong generalizability, serving as a versatile approach for tool invocation for various domains.

Acknowledgements

This research was supported by grants from the grants of Provincial Natural Science Foundation of Anhui Province (No.2408085QF193), USTC Research Funds of the Double First-Class Initiative (No. YD2150002501), the National Key Research and Development Program of China (Grant No. 2024YFC3308200), the National Natural Science Foundation of China (62337001), the Key Technologies R & D Program of Anhui Province (No. 202423k09020039) and the Fundamental Research Funds for the Central Universities (No. WK2150110032).