VRD-IU: Lessons from Visually Rich Document Intelligence and Understanding

1 Introduction

Visually Rich Document Understanding (VRDU) has garnered significant attention across various domains, including medical [Ding et al., 2024b], receipts [Park et al., 2019; Huang et al., 2019], financial [Stanisławek et al., 2021], and educational [Wang et al., 2021] fields, becoming a pivotal approach for recording, preserving, and sharing information. Recent advancements in deep learning, particularly self-supervised pretrained transformers [Huang et al., 2022; Tang et al., 2023; Xu et al., 2021] and large language models [Xue et al., 2024; OpenAI, 2024; Liu et al., 2024], have demonstrated notable success in this domain. However, form-like documents present unique challenges due to their involvement of multiple parties (designers, deliverers, and fillers), complex layouts, and higher uncertainties.

Figure 1: Digital, Printed and Handwritten Form Samples from Form-NLU dataset.

These factors create significant barriers to directly applying existing methods to practical tasks [Ding et al., 2024c; Ding et al., 2024a].

To address the challenges caused by form-like documents, we introduce the VRD-IU Competition, which focuses on extracting key information from multi-format forms within the Form-NLU dataset [Ding et al., 2023], encompassing digital, printed, and handwritten documents, as illustrated in Figure 1. The competition aims to tackle the challenges associated with the diverse and complex nature of such documents, which often involve multiple stakeholders and contain essential information that is difficult to extract. By offering two distinct tracks tailored to different skill levels, it accommodates participants ranging from newcomers to seasoned professionals in the field of deep learning. This initiative not only fosters innovation in document understanding and accelerates advancements in information extraction techniques but also aims to engage a broader community, encouraging innovators to contribute to the evolution of efficient information analysis methodologies.

More than 20 teams participated in the VRD-IU competition, with the top 3 teams in Track A and the top 2 teams in Track B submitting their code and abstract papers to share their proposed solutions. Various methods were proposed, focusing on achieving more comprehensive document understanding. In Track A, the methods mainly focused on acquiring more representative entity features, while in Track B, end-to-end frameworks were proposed to enable the framework to understand both the semantic and structural aspects of form images to extract and locate the relevant information.

The proposed methods achieved new state-of-the-art performance and paved the way for future work in complex form understanding direction.

The contributions of this paper are as follows:

1) To provide a redefined task definition for the Form-NLU dataset, focusing on Key Information Localization (Track-B), which involves both extracting and localizing target information from input form images.

2) To introduce the state-of-the-art approaches proposed by competition participants, along with a qualitative analysis of these methods.

3) To offer key insights into future directions for VRDU understanding, highlighting potential areas for improvement and innovation.

2 VRD-IUCompetition Task Definition

The adopted dataset for the VRD-IU competition is derived from the Form-NLU dataset. Unlike the original tasks, which emphasize both form structure and content understanding, this competition specifically targets the extraction of key information from complex form documents. To address varying levels of task complexity, two distinct tracks are introduced. Track A aligns with Form-NLU Task B, focusing on retrieving key information from provided semantic entities within the document. In contrast, Track B presents a more challenging scenario, offering only the form image and query without any prior information about the document structure. The detailed definitions of these tracks are provided to support future research in this area.

Track A: Form Key Information Extraction 1 Track A involves developing deep learning-based retrieval models to extract specific form components in response to a given key query. Human-annotated bounding box coordinates representing the semantic entities of input form documents are provided. The task requires accurate identification and localization of the target entity based on the query. Performance is evaluated using the F1-Score, following the evaluation methodology defined in Form-NLU Task B.

Track B: Form Key Information Localization 2 Track B focuses on designing an end-to-end framework to predict bounding box coordinates of form components directly from input document images based on a given key query. Unlike Track A, ground truth bounding boxes for semantic entities are not provided; the inputs consist solely of form images and query keys. Evaluation is based on the Mean Average Precision (MAP) of the predicted bounding boxes, reflecting both accuracy and localization effectiveness.

1 The dataset used for Track A can be found via https://www. kaggle.com/competitions/vrd- iu2024- tracka

2 The dataset used for Track B can be found via https://www. kaggle.com/competitions/vrd- iu2024- trackb

3 Methods

3.1 Methods for Track-A

To effectively retrieve key information from domain-specific semantic entities based on user queries, several solutions leverage enhanced document understanding through pretrained transformers. These approaches incorporate advanced components and strategies hierarchical decomposition, token classification, and multimodal feature fusion, to better comprehend the structure and semantics of forms, enabling more accurate and efficient extraction of target information.

Key Information Extraction with Hierarchical Layout Decomposition, proposed by Choi et al. (Team rb-ai, Top-1 Team in Track-A), utilizes prior template knowledge from the Form-NLU dataset to segment form images into top, middle, and bottom sections. Then, section-specific KIE models are then applied for key information extraction. To address misclassification issues within each section, heuristic postprocessing techniques are employed. The approach uses a pretrained transformer-based YOLOv8 as an object detector for section segmentation, while LayoutLMv3 [Huang et al., 2022] is fine-tuned as the KIE backbone for each segmented section.

Redefining Information Extraction from Visually Rich Documents as Token Classification is introduced by Song et.al (Team vipski, Top-2 Team in Track A) to convert the entity retrieving task as a sequence labelling task with maintaining the original resolution and aspect ratios of document images, the approach reduces visual distortion, enhancing model accuracy. After fine-tuning LayoutLM-v3 with a token classifier, the proposed method achieve cutting-edge performance.

Transformer-Based Visual Feature and Textual Feature Fusion Model for Document Key Information Extraction, proposed by Kim et al. (Team gcu, Top-3 Team in Track A), aims to enhance document semantic entity representation. RoI visual and textual features are separately processed through a transformer-based vision-language encoder. A cross-encoder then strengthens the correlations between questions and entities. The combined textual and visual representations are fed into a classifier for final information retrieval.

3.2 Methods for Track B

Different from Track A providing the bounding box coordinate of each document semantic entity, Track B requires the model can detect the exact object (semantic entities) from the document image. To realize this, except for the well-designed object detection approaches, some domain-adaptation strategies are adopted.

Key Information LocalizationWith Hierarchical Ensemble is introduced for key information localization by Choi et al. (Team rb-ai, Top-1 Team in Track-A). The approach uses YOLO [Redmon et al., 2016] and RT-DETR [Zhao et al., 2024] models as baselines and integrates their predictions through a global ensemble and a layout-specific ensemble. Additionally, a layout-based post-editor refines predictions using text recognition and layout-specific detectors to correct mispredictions.

Enhancing Document Key Information Localization Through Data Augmentation is proposed by Dai et al. (Team chatdy, Top-3 Team in Track-A), which divides the whole workflow to document augmentation phase and an object detection phase. The augmentation phase uses the Augraphy library to mimic the appearance of handwritten documents from digital ones, emplointoying techniques like InkBleed, Letterpress, and JPEG compression to simulate handwritten characteristics. The object detection phase utilizes pretrained VRDU models like DiT [Li et al., 2022] and LayoutLMv3 with Faster R-CNN [Ren et al., 2015] and Mask R-CNN [He et al., 2017] frameworks. The key innovation lies in the augmentation strategy, which enhances the model's ability to generalize to handwritten documents, even when trained only on digital data.

4 Competition Results

The VRD-IU competition attracted significant participation from teams employing diverse methodologies to tackle the key information extraction and localization challenges. Track A and B's evaluation results reveal notable trends in model performance and methodological efficacy.

Track A: Form Key Information Extraction

The results of the top-performing teams in Track A are shown in Table 1. The results demonstrate that pretrained transformer-based approaches, enhanced with hierarchical decomposition and multimodal feature fusion, are highly effective in extracting semantic entities from form documents.

Track B: Form Key Information Localization

Performance in Track B was evaluated using Mean Average Precision (MAP), reflecting both detection accuracy and localization effectiveness, shown in Table 2. Track B posed a greater challenge due to the lack of predefined bounding box annotations. The results indicate a clear performance gap compared to Track A, underscoring the complexity of end-toend document understanding directly from form images.

5 Key Insights

The competition provided valuable insights into document understanding tasks, particularly within visually rich document (VRD) contexts:

1. Hierarchical Decomposition and Post-Processing Improve Accuracy Track A results highlighted that segmenting form structures into hierarchical components significantly enhances extraction accuracy. Techniques such as section-wise key information extraction (as seen in rb-ai's approach) help mitigate structural misclassification and boost precision.
2. Pretrained Transformers Excel in Structured Document Tasks Methods leveraging LayoutLMv3 and other pre-

trained vision-language transformers outperformed conventional models, reaffirming the effectiveness of multimodal pretraining for VRD tasks. Fine-tuning on domain-specific data further improved entity extraction capabilities.

1. Object Detection Remains a Challenge for Form Localization Track B results highlighted the difficulty of localizing key information without predefined bounding boxes. Despite the effectiveness of YOLO and RT-DETR models, the lower MAP scores indicate that further advances in end-to-end document understanding are needed.
2. Data Augmentation Enhances Generalization Augmenting digital forms with synthetic effects improved model robustness for handwritten document retrieval (e.g., ChatDY's approach). This suggests that document variability must be carefully considered in future dataset preparation.
3. Ensemble Approaches Provide Performance Gains Models integrating multiple object detection techniques (such as rb-ai's hierarchical ensemble strategy) showed superior performance, indicating that fusing diverse model outputs can enhance accuracy in complex layouts.

6 Conclusion

The VRD-IU competition served as an effective benchmark for evaluating state-of-the-art approaches in key information extraction and localization within form documents. Overall, the VRD-IU competition has accelerated research in visually rich document understanding by providing a well-defined benchmark and fostering innovation in key information extraction and localization. The insights gained will be instrumental in guiding the development of more robust, efficient, and generalizable AI-driven document processing systems.

Ethical Statement

There are no ethical issues.

Acknowledgments

Thanks for the support from Google Research, United States.

Contributors

We would like to acknowledge the invaluable contributions of the authors across all competition winners in IJCAI 2024 VRD-IU Competition:

• Track A - Team rb-ai (Rank 1): Key Information Extraction with Hierarchical Layout Decomposition: Hongjun Choi (Teamreboott Inc., South Korea), Jongho Lee (Teamreboott Inc., South Korea; Pukyong National University, South Korea), and Jaeyoung Kim (Teamreboott Inc., South Korea).
• Track A - Team vipski (Rank 2): Redefining Information Extraction from Visually Rich Documents as Token Classification: Jonghyun Song and Eunyi Lyou (Graduate School of Data Science, Seoul National University).
• Track A - Team gcu (Rank 3): Transformer-Based Visual Feature and Textual Feature Fusion Model for Document Key Information Extraction: Wooseok Kim, Juhyeong Kim, Sangyeon Yu, Gyunyeop Kim, and Sangwoo Kang (School of Computing, Gachon University).
• Track B - Team rb-ai (Rank 1):Key Information Localization With Hierarchical Ensemble: Hongjun Choi (Teamreboott Inc., South Korea), Jongho Lee (Pukyong National University, South Korea), and Jaeyoung Kim (Teamreboott Inc., South Korea).
• Track B - Team chatdy (Rank 2):Enhancing Document Key Information Localization Through Data Augmentation: Yue Dai (The University of Western Australia).

Their dedication and innovative research have significantly advanced the field of key information extraction and visually rich document understanding.