MoleculeMiner: Extracting and Linking Molecule Figures with Tabular Metadata

1 Introduction

While structured extraction from unstructured documents in the chemical domain is not new, most of these systems have modality and input format limitations. Some recent systems for chemical structure recognition (CSR) include DECIMER [Rajan et al., 2020] that used a CNN based encoder-decoder architecture to directly predict SMILES from a molecule image. MolScribe [Qian et al., 2023], which MoleculeMiner's parsing is based off of, used a transformer based encoderdecoder model but was limited in performance due to being trained on synthetically generated data. MolGrapher [Morin et al., 2023] used a different approach by first using a CNN to predict the atoms and then use a graph neural network (GNN) to predict the graph structure of the molecule. This was then converted to a SMILES representation. ChemScraper [Shah et al., 2024] on the other hand used two different parsing techniques for born-digital (rules) and image-based (neural) to improve accuracy if the PDF has embedded graphical elements for the molecule figures. MoleculeMiner aims to improve upon the existing MolScribe CSR system by adding a novel positional embedding technique and addition of new forms of data augmentation for CSR to be robust to diagrams extracted from patents as well as journals.

Extracting tabular data in the chemical domain has always been a challenge due to the large variations in table styles found. Further, these tables often don't contain the chemical structure, and instead have a text identifier to the compounds. ChemTables [Zhai et al., 2021] tried an approach by trying to classify the type of table first by populating a table type ontology from Reaxys [Reaxys, 2009]. ChemDataExtractor [Swain and Cole, 2016] used a layout classifier to identify tables from other objects such as images and text blocks based on a heirarchy of rules. The segmented tables were then parsed row-wise based on embedded characters in the PDF. Thus, it could only support born-digital documents as their base extraction system was based on PDFBox [The Apache Software Foundation, 2012]. Modern approaches introduce using vision based multi-modal Large Language Models (LLM) for zero-shot extraction of table data given a specific schema through prompt engineering. ChatExtract [Polak and Morgan, 2024] used a hierarchy of prompts to extract chemical data from tables in a JSON format. MoleculeMiner attempts to automate the table extraction and linking using carefully constructed prompts to identify chemically relevant tables.

Co-referencing or entity-linking different modes of information sources from a document in the chemical domain is an equally challenging task due to variations in document generation methods and structure. [Zhang et al., 2023] used a BiLSTM based Named Entity Recognition (NER) model to extract relevant chemical text and a CNNmodel to performOptical Character Recognition (OCR) on document tables to link material names in text with associated units and composition data in tables. However their table schema that were filtered had to adhere to a fixed expected schema. ChemSchematicResolver [Beard and Cole, 2020] linked R-group definitions in text to diagrams containing open group substituent. OpenChemIE [Fan et al., 2024] is an end-to-end system that used a table detection method and identifying relevant ones though table headers to replace R-groups in diagrams. Our system can identify links between any form of structured tables and diagrams as long as there is a reference number written in its vicinity.

Figure 1 shows the overall pipeline of MoleculeMiner. When a user provides a PDF document (born-digital or scanned), the system runs it through a multi-stage pipeline that (1) Detects, extracts and parses Drawn Molecule Diagrams into a Simplified Molecular Input Line Entry System (SMILES) [Weininger, 1988] string (Chemical Structure Recognition (CSR)) (2) Detects and extracts all data from tables (Table Extraction) (3) Co-references molecule property data from tables and links them to parsed molecules (Entity Linking).

2 SystemDescription

2.1 Molecule Detection and Parsing

The detection and parsing system first identifies the regions where the molecules exists, then parses the individual molecules through a transformer based encoder-decoder system to predict the graph structure of the molecule. The graph is then post-processed to generate the canonical SMILES string. This representation is useful for a variety of reasons - applicability in molecular downstream tasks, encoding of chemical properties such as chirality and double-bond geometry (stereochemistry) and easy interconversion to other formats such as MOLFile, DeepSMILES [O'Boyle and Dalke, 2018] and InChI [Heller et al., 2015] [Heller, 2014].

Molecule Diagram Detection: PDF pages are processed through YOLOv8 [Jocher et al., 2023] to segment the molecule regions. CLEF-IP2012 [Piroi et al., 2012] [Dey and Zanibbi, 2021] dataset consisting of 1242 pages and 419 pages for train and test respectively containing molecule diagrams. Due to limitation in availability of open source training data for segmentation, we avoid using data intensive options like vision transformer for this stage.

Molecule Diagram Parsing: Our parsing model, MolScribeV2, improves upon the original MolScribe [Qian et al., 2023] in three major areas - enhanced positional embeddings, dataset size and special augmentation. Figure 2 shows the additional positional embedding method where at each block of the SWIN transformer encoder [Liu et al., 2021], an additional learnable positional embedding is created from the binary mask of graphics pixels that forces attention values in the transformer layers to distinguish between valid and invalid pixels. Compared to MolScribe which was trained on 1 million PubChem [Kim et al., 2022] molecules, 5 million molecules were used to train MolScribeV2 with data from PubChem, Zinc250k [Akhmetshin et al., 2021], ChemBL [Mendez et al., 2019], MOSES [Polykovskiy et al., 2020]. Finally, existing data augmentation methods were combined with two new techniques to improve performance on real world data. Margin cropping or adding was a method used to cut or append margins (upto 10% of height or width of the image) at any side of the original to simulate real world examples of images arising from imperfect segmentations. Document degradation was another augmentation that added random amounts gaussian and salt and pepper noise to the images to simulate molecule regions from scanned PDFs. Table ?? shows the comparison of original version to various improvements on a set of two real-world patent PDFs. 1832 molecules were manually collected along with their groundtruth SMILES. This set was collated with expert annotated SMILES to produce the Page → Location → Molecule → SMILES examples from real world documents. This is the first such collection made to our knowledge and will be made available. We do not report benchmark comparisons due to space constraints and those not reflecting real world extracted molecules.

2.2 Molecule and Table Data Linking

The process involves three main stages. (1) Finding the reference number written in the proximity to each drawn molecule (2) Extracting all tables found in the PDF and filtering only the tables containing a column with reference numbers (3) Merging drawn molecules with their associated table metadata based on common reference numbers.

References to drawn molecules were found using the docTR [Mindee, 2021] OCR suite. After filtering out all words that either were not numeric or alphanumeric or were inside a molecule region, the rest of the words were chosen as candidate sets for attaching to each individual molecule. Proximity-based matching included spatial constraints such as reference numbers can only be to the bottom or right of a molecule, each reference number instance can only be used without replacement and have to fall within certain distance based on the longest side of molecule region. This approach was found to be most consistent across different styles of documents.

Table Detection and Extraction was performed through GPT-4o [et al., 2024] through a series of prompts. The prompts were carefully designed to sequentially extract the table label, table headers and the row metadata. Additional algorithms were designed to join multi-page tables together, process tables with no table headers and filter out columns where the molecule is drawn inside the table itself.

Linking molecules with their associated metadata involved collating reference numbers found near drawn molecules with reference numbers in tables. In our work, we found reference column headers could be named in a different ways and specific keyword list was designed to account for the variations.

The approach taken in this paper for co-referencing improves previous systems in various ways. Multi-page tables can be successfully joined together, forming a unified coherent table for linking regardless of table schema change between pages. Each molecule can be linked to metadata obtained from more than one table. Any type of metadata including multi-line metadata can be successfully linked without the need of constraining to very specific table schemas.

3 User Interface

To facilitate user adoption of Molecule Miner, we have created a user interface allows users to run analyses with little effort, including those lacking programming experience (see video accompanying paper). The user can upload a PDF of interest and, optionally, their OpenAI API key if they want table extraction as well. Once complete, the user is served an interactive display of their document with clickable red boxes indicating an identified molecule. Upon clicking each red box, the user can see information about the molecule like its SMILES and the model confidence. Figure 3 shows available metadata collated from the entire PDF for a specific molecule displayed to the user. Furthermore, a user can directly download the full PDF result as a CSV file that can be directly used in downstream applications like molecule modeling, property prediction etc.

4 Conclusion

MoleculeMiner uses specialized extraction and coreferencing methods, linking parsed diagrams with its pharmacologically relevant data mentioned in tables. This is enabled by our robust molecule diagram extraction and improved parsing along with the usage of Large Language Models to detect and parse tables. Automated linking of relevant table data with molecules allows important molecule properties to be easily accessible to be used in other downstream tasks for cheminformatics. The webapp exposes this functionality in an easy-to-use interface through which users can get information from both digital and scanned PDFs.

Acknowledgments

We would like to thank our colleagues in DS-ML team at Insitro for their contributions in making this project including Patrick Conrad, Srinivasan Sivanandan, Kent Gorday, Tomasz Danel, and Benson Chen for their thoughts and ideas and infrastructure help for this project.