Property Enhanced Instruction Tuning for Multi-Task Molecule Generation with Large Language Models

1 Introduction

Large language models (LLMs) such as GPT-4 [OpenAI, 2023], Gemini [Comanici et al., 2025] and DeepSeek [Guo et al., 2025] have revolutionized the landscape of artificial intelligence and natural language processing, allowing machines to understand and generate human language with remarkable fluency and coherence. Based on encoded world knowledge [Petroni et al., 2019] and powerful instructfollowing [Zhang et al., 2023] capabilities of LLMs, recent work has successfully used LLM for molecular-related tasks, achieving promising results [Cao et al., 2025].

∗Corresponding author.

Despite their success, as illustrated in Figure 1 (a) and (b), LLMs still struggle with generating molecules under strict property constraints. Even specialized molecular translation models, such as MolT5 [Edwards et al., 2022], fail in these tasks (see Figure 1 (c)). While these models effectively capture relationships between molecular text and structure, they lack sufficient understanding of molecular properties, limiting their ability to incorporate property constraints in prompts. This shortcoming restricts their practical utility in applications like drug discovery [Zhavoronkov, 2018].

The challenges in addressing such tasks mainly lie in three aspects: (1) Existing studies have revealed the limitations of LLMs in understanding molecular representations [Grisoni, 2023], making it difficult to handle tasks requiring precise molecular property comprehension; (2) While there are some known SMILES-property pairing data, it often remains limited to predicting a single property and lacks datasets that cover a wide range of properties [Wu et al., 2018]. Moreover, most of these datasets do not include precisely described textual data, making it challenging to identify accurate trimodal data pairs; (3) To our knowledge, there are no suitable datasets or evaluation methods exist for multi-constraint molecule generation using LLMs, which challenges the standardization and assessment of such tasks [Elton et al., 2019].

To address these challenges, we propose a framework called PEIT (Property Enhanced Instruction Tuning) to generate multi-modal molecular instruction datasets in bulk, aiming to enhance capabilities of LLMs in multi-task molecule generation. Using the PEIT framework, our pre-trained model can handle general tasks (e.g., molecule captioning [Edwards et al., 2022]) and property-related tasks such as property prediction [Chang and Ye, 2024]. This makes it suitable for constructing data to evaluate multi-constraint molecule generation capabilities and for serving as instruction tuning data to improve existing open-source LLMs.

The overall structure of the proposed PEIT framework is shown in the left of Figure 2. Specifically, it consists of two components: (1) We pre-train a model called PEIT-GEN through multi-modal representation alignment, which integrates text-based (molecular descriptions), structure-based (SMILES), and property-based (property-value pairs) information to generate diverse unstructured text, sequence, and property data; (2) By using the synthesized instruction data, we fine-tune open-source LLMs and develop PEIT-LLM, which can be applied to various molecule generation tasks mentioned above, including our proposed multi-constraint molecule generation.

Experimental results demonstrate that our pre-trained PEIT-GEN achieves competitive or better results in molecule captioning tasks, comparing to a variety of biomolecular models including MolT5 [Edwards et al., 2022], BioT5 [Pei et al., 2023], GIT-Mol [Liu et al., 2024], MolXPT [Liu et al., 2023a], MolCA [Liu et al., 2023b], and Text+ChemT5 [Christofidellis et al., 2023]. Additionally, PEIT-LLM based on LLaMa3.1-8B [Dubey et al., 2024] exhibits superior performance compared to specialized models MolInstructions [Fang et al., 2023] and general-purpose LLMs including LLaMa3 [Dubey et al., 2024] and Qwen2.5 [Yang et al., 2024] in molecular property prediction and our newly proposed multi-constraint molecule generation tasks.

2 RelatedWork

Molecule Generation. Molecule generation tasks fall into two categories: (1) text-based molecule generation that uses textual descriptions to generate molecules that match the given description [Liu et al., 2023a; Liu et al., 2024]. MolT5 [Edwards et al., 2022] was the first proposed to realize translation between textual description and molecular SMILES. BioT5 aims to enhance molecular understanding by incorporating protein. They also perform molecule captioning, which is equivalent to the inverse task of text-based molecule generation. (2) property-guided molecule generation is the inverse process of molecular property prediction, where molecules are generated based on specific biochemical property constraints. Notably, SPMM [Chang and Ye, 2024] was the first to establish a connection between 53 biochemical properties and SMILES sequences, making multi-constraint molecule generation possible. However, few existing models can simultaneously perform text-based or multi-constraint molecule generation and molecule captioning.

Molecular Property Prediction. Deep learning models have been developed for molecular property prediction each with their own advantages and limitations. Transformer-based models design attention mechanism to capture contextual contexts from large-scale SMILES sequences [Ross et al., 2022]. The molecular graph can be directly obtained from SMILES sequences via RDKit [Landrum and others, 2013]. Graph-based models develop diverse graph neural networks to learn differentiable representations [Wang et al., 2022]. However, these methods ignore the potential that incorporating textual knowledge enables to realize new drug design objectives. Recently, a novel molecular pre-trained model named SPMM [Chang and Ye, 2024] that extends the application of multimodal pre-training approaches by aligning molecular structures and biochemical properties. This paper extends the multimodal pre-training to patterns of textsequence-property triplets, which is defined flexibly by LLMunderstandable textual prompts.

Instruction Tuning. Constructing specialized instruction datasets is an effective way to enable LLMs to better perform molecular-related tasks. For instance, Mol-Instructions [Fang et al., 2023] provides a large-scale biomolecular dataset tailored for LLMs, covering a variety of instructions involving small molecules, proteins, and biomolecular texts. ChemDFM [Zhao et al., 2025] advances this paradigm by creating a broader dataset spanning molecular structures, reactions, and properties. Its two-stage training-domain pretraining followed by instruction tuning-enhances the model's chemical understanding and reasoning capabilities. More recently, GeLLM3O [Dey et al., 2025] introduced MuMOInstruct, a high-quality dataset focused on multi-property molecule optimization, demonstrating strong generalization across diverse tasks. Despite these advances, generating reliable and scalable molecular instruction data remains a key challenge, particularly for open-source models.

3 Method

3.1 Overview of PEIT Framework

The overview of PEIT framework is shown in Figure 2 (left), which consists of PEIT-GEN and PEIT-LLM. In PEITGEN, we generate a large number of "SMILES-text" and "SMILES-property" pairs to serve as multi-modal data. Then we design multiple multi-modal alignment objectives to pretrain PEIT-GEN. In PEIT-LLM, by using the pre-trained PEIT-GEN, we can predict a large number of triplets to generate more diverse SMILES inputs, and then construct diverse instruction data based on template filling. By utilizing the synthesized instruction data, PEIT-LLM enables the supervised fine-tuning of open-source LLMs including LLaMa [Dubey et al., 2024] and Qwen [Yang et al., 2024], enhancing the capabilities for multi-task molecule generation.

3.2 Pre-training of PEIT-GEN

The pre-training stage of PEIT-GEN is shown in the right of Figure 2. For a given molecule, different representations offer unique and complementary features, which are crucial for comprehensive molecule understanding. PEITGEN aims to integrate information from three modalities simultaneously, including textual information T (text), molecular structure S (SMILES), and biochemical properties P (property-value). Such ability can help synthesizing sufficient instruction data for further enhancing the ability of LLMs. In particular, PEIT-GEN consists of three Transformer encoders Enct , Encs , Encp and two decoders Dect , Decp , and we design different training objectives to align features from different modalities.

Cross-modal Representation Matching. Following SPMM [Chang and Ye, 2024], we leverage pre-trained models SciBERT [Beltagy et al., 2019] as trainable Enct for encoding textual data, BERT [Devlin et al., 2019] as Encs and Encp for encoding SMILES and properties. Then we obtain feature representations across all three modalities, establishing the foundation for feature alignment.

We propose cross-modal representation matching to align the representations from different perspectives by the same molecule. In particular, we introduce the SMILES-text matching loss Lst match and the SMILES-property matching loss Lsp match , which serve as objectives for training the encoders. In this way, the model can effectively learn cross-modal relationships and improve performance in multi-modal tasks by aligning the feature spaces. The matching loss is calculated as follows:

where yst match and ysp match are labels as 0 or 1, indicating whether the corresponding SMILES-text or SMILES-property pairs are matching. Enc(·) indicates the representation of the data (i.e., [CLS] token of Transformer encoder), ⊕ is the concatenation operation, and MLP(·) is the trainable multi-layer perception. The encoders are optimized by the cross-entropy loss ℓCE using the given data from different modalities.

Multi-modal Contrastive Learning. The representation matching can be viewed as an explicit 2-way classification training. We further utilize contrastive learning to directly enhancing the representation by pulling semantically close neighbors together and pushing apart non-neighbors from data of different modalities. To calculate the similarity between the encoded features of different modalities, we extract the encoded features and then compute the instance-level similarities through the inner product:

where MLPs , MLPt and MLPp are multi-layer perceptions applied to SMILES, text, and property representations, respectively. Then, for the given SMILES S, text T , and property P, we compute the cross-modal batch-level similarities as follows:

where M and N represent the total number of texts and property in the batch of data pairs, respectively. τ is the temperature controlling the sharpness of the similarity. The intramodal similarities ss2s, sp2p, and st2t can be computed in similar manners.

Based on the cross-modal and intra-modal batch-level similarities, the contrastive loss is formulated by calculating the cross-entropy according to one-hot encoded similarity vectors y, where the value is 1 for pairs derived from the same molecule or 0 for all other combinations:

Cross-modal Causal Language Modeling. To further strengthen the model's capability in molecule captioning, we employ the causal language modeling (CLM) to enhance the model performance on text generation. Specifically, we design decoders to generate subsequent property and textual description sequences, under the guidance of SMILES features through cross-attention as show in Figure 3.

By introducing the SMILES features in attention layers for CLM training, the cross-modal CLM loss Lst CLM and Lsp CLM are computed as follows:

where Prob is the conditional probability to predict the word w(i) j in the vocabulary, N is the total number of samples, n is the index of current words in each sample, ˜w(i) :j is the sequence from begin to the j-th word in the i-th sample, θt and θp are the trainable parameters in two decoders.

Training. The overall training objective for pre-training PEIT-GEN is to minimize the sum of all three types of losses across three modalities:

where we conducted a hyperparameter search for α and β, as shown in Table 1 in Appendix A. Based on the experimental results, we follow the setting in SPMM[Chang and Ye, 2024] and adopt a ratio of α : β = 1 : 5 to balance the loss terms.

3.3 Instruction Tuning for PEIT-LLM

Template Filling. The pre-trained PEIT-GEN offers unstructured data in the format of "text-SMILES-properties" (i.e., text-structure-property) triplets, which are stored in CSV files containing text, molecular structures, and information on 53 molecular biochemical properties. To obtain more taskspecific data and to adapt to the strong instruction-following abilities of LLMs, we design templates for different downstream tasks, as shown in Figure 1 of Appendix B. For textbased molecule generation as example, we fix a general question format and then extract molecular descriptions from unstructured data to fill the pre-defined template, resulting in a natural question as instructions. The SMILES from unstructured triplets is used as the desired response. In this way, we can generate diverse task-specific instruction data in bulk for subsequent instruction-tuning.

Multi-constraint Molecule Generation Task. Molecule generation often requires to be conducted under multiple constraints rather than a single condition. In this work, we propose a new task to assess molecule generation through a variety of descriptors, by comparing the alignment between the generated molecules and specific criteria to evaluate the generative performance of LLMs. By using the large-scale unstructured data generated by PEIT-GEN, we can effectively synthesize sufficient data for evaluation. Specifically, we follow SPMM [Chang and Ye, 2024] given the vast number of molecular attributes and the complex combinations thereof, analyzing their impact on results across different molecular counts poses a signifi- cant computational challenge due to resource limitations. To address this, we selected representative ADMET [Fu et al., 2024] properties-BalabanJ, MolLogP, ExactMolWt, QED, and TPSA-that capture molecular topology, electronic characteristics, and steric effects, while exhibiting low mutual correlation. Based on template filling, the predicted multi-property values are used to construct data for multi-constraint molecule generation. We employ instruction tuning to guide the LLM in generating molecules, and use RDKit [Landrum and others, 2013] to calculate the actual property values. RMSE and R2 are then used to compare these values against the constraints, enabling a systematic evaluation of the LLM's performance in multiconstraint molecule generation tasks.

Supervised Fine-tuning. We select LLaMa3.1-8B [Dubey et al., 2024] and Qwen2.5-7B [Yang et al., 2024] as base LLMs. We then perform standard SFT [Ouyang et al., 2024] by using the "instruction-response" pairs. In practice, we construct totally 1 million instruction data of four different tasks (i.e., molecule captioning, text-based molecule generation, property prediction, and multi-constraint molecule generation) from 200k unstructured "text-SMILES-properties" triplets obtained by PEIT-GEN.

3.4 Comparing PEIT-LLMwith Biomolecular Models and Large Language Models

Table 1 shows a comparison of our PEIT-LLM with existing pre-trained models and general LLMs on multiple molecular generation tasks. For most of the pre-trained models such as MolT5 and BioT5, they focus on molecule captioning and text-based molecule generation, which can not handle property-related tasks. SPMM is a specialized model for property prediction. However, it lacks generation ability due to the lack of textual descriptions. Current LLMs such as LLaMa and Qwen show strong performance on general NLPbased tasks through conversations or instruction-following. However, these general LLMs still have limitations in tasks related to molecule generation due to a lack of molecular knowledge. In contrast, through fine-tuning on diverse instruction data with rich molecular knowledge, PEIT-LLM can perform multiple molecule generation tasks simultaneously.

4 Experiments

4.1 Experimental Setup

Dataset. During the pre-training of PEIT-GEN, we extract 480k molecular SMILES from the ZINC dataset [Irwin et al., 2012] and generate corresponding SMILES-text pairs using MolT5. Meanwhile, we compute 53 biochemical properties for each molecule using RDKit, forming 480k "text-SMILES-property" triplets for training. We then further fine-tune the model on the CHEBI-20 [Edwards et al., 2021] dataset to enhance its performance on humanannotated data.

For pre-training PEIT-LLM, we utilize the 200k tri-modal data generated by PEIT-GEN and employ template filling to generate 200k instruction data for each downstream task. For molecular property prediction, we select two biochemical properties with distinct differences for evaluation, generating 200k instruction data for each property. Finally, we obtain a total of 1000k instruction data across four tasks for SFT. Similar to PEIT-GEN, molecular property prediction tasks on PEIT-LLM can be validated by RDKit on CHEBI-20 dataset.

To evaluate PEIT-GEN and PEIT-LLM, we follow MolT5 by using CHEBI-20 and MoleculeNet dataset [Wu et al., 2018], with the standard splition into training, validation, and test sets with an 8:1:1 ratio. All property values are verified via RDKit. See details in Appendix C.

Baseline Models. We compare our model, PEIT-GEN and PEIT-LLM, against three types of baselines as follows: Baselines on molecule caption such as MolT5 [Edwards et al., 2022], BioT5 [Pei et al., 2023], MolCA [Liu et al., 2023b], Text+Chem-T5 [Christofidellis et al., 2023], GIT-Mol [Liu et al., 2024]. Baselines on moleucular property prediction such as SPMM [Chang and Ye, 2024], D-MPNN [Yang and others, 2019], PretrainGNN [Hu et al., 2019], GROVER [Rong et al., 2020], ChemRL-GEM [Fang et al., 2022]. Baselines of LLMs such as LLaMa3 [Touvron et al., 2023], GPT3.5turbo [OpenAI, 2023], Mol-Instructions [Fang et al., 2023], Qwen2.5 [Yang et al., 2024], BioMedGPT [Zhang et al., 2024], ChatGLM [GLM et al., 2024], LLaSMol [Yu et al., 2024],ChemDFM [Zhao et al., 2025], InstructMol-GS [Cao et al., 2025], Gemini [Comanici et al., 2025], Details and evaluation metric are in Appendix D and E, respectively.

Implementation Details. We pre-train PEIT-GEN for 20 epochs using a batch size of 16, temperature τ = 0.07, and momentum 0.995 with the AdamW optimizer [Loshchilov, 2017]. Fine-tuning is then performed on the CHEBI-20 training set for 200 epochs with a learning rate of 5e-4. For supervised fine-tuning of PEIT-LLM, we use the LLaMaFactory [Zheng et al., 2024] framework with LoRA [Hu et al., 2022] for 6 epochs, a batch size of 3, and learning rate of 5e-5. The total parameter count of the three encoders and two decoders is 533.8M, with an additional 2.2M parameters for other components. Experiments are conducted on NVIDIA 4090 GPUs with 24GB memory.

4.2 Comparing PEIT-GENwith Pre-trained Biomolecular Models

Molecule Captioning. Results on molecule captioning using CHEBI-20 dataset are shown in Table 2. Our model demonstrates superior performance in generating high-quality and relevant molecular caption. PEIT-GEN achieved the best results in METEOR and ROUGE, and the second-best performance in BLEU-4. Compared to BioT5 which performs the best in BLEU, our approach requires significantly less data. This indicates that using domain-specific models to generate paired data for pre-training is more efficient than singlemodality pre-training.

Molecular Property Prediction. We evaluate the generalization ability of PEIT-GEN on the MoleculeNet benchmark [Wu et al., 2018] using four widely adopted tasks. As shown in Table 3, PEIT-GEN outperforms specialized models such as MolFormer [Ross et al., 2022] and ChemRLGEM [Fang et al., 2022] on the Clintox dataset. Despite using less pre-training data, it remains competitive on other subsets. To further demonstrate its predictive strength across 53 molecular properties, we present a relative difference analysis in Figure 2 of Appendix F, highlighting PEIT-GEN's strong generalization in property prediction.

4.3 Comparing PEIT-LLMwith LLMs

Molecule Captioning. As shown in the top of Table 4, the comparison results show that our model outperforms generalpurpose Qwen-2.5 and LLaMa3.1 as well as Mol-Instructions and BioMedGPT, which were trained using a biochemical information instruction dataset for SFT. PEIT-LLM achieved the second-best performance on the ROUGE metric and demonstrated competitive results compared to InstructMol- GS, which was trained solely on the CHEBI-20 dataset and has a similar parameter scale as our base model.

Text-based Molecule Generation. Results on the CHEBI20 test set are presented at the bottom of Table 4. PEITLLM outperforms all baselines on numerical metrics, including BLEU, Levenshtein Distance, and fingerprint similarities based on MACCS, Morgan, and RDKit. Although MolInstructions achieves the highest Validity score, the results demonstrate that PEIT-LLM, after multi-task instruction finetuning, effectively captures key molecular structures and corresponding textual representations. Case study in Table 5 of Appendix G further supports these findings and indirectly validates the quality of the data generated by PEIT-GEN.

Molecular Property Prediction. For single-property prediction, due to the large number of available properties, we select two representative examples: ExactMolWt, which typically has large numerical values (100 to 1000), and MolLogP, with smaller values (-5 to 10), as shown in Table 5. The results show that PEIT-LLM consistently outperforms other LLMs in predicting these biochemical properties, demonstrating strong sensitivity and adaptability to molecular property scales. This highlights the effectiveness of multi-task SFT in enhancing LLMs' understanding of molecular characteristics and further validates the quality and reliability of our molecular property instruction dataset. A case study is provided in Table 6 of Appendix G for further illustration.

Multi-constraint Molecule Generation. Results for our proposed multi-constraint molecule generation task is shown in Table 5. PEIT-LLM surpasses baselines by large margin in both RMSE and R2 metrics. Case study is provided in Table 7 of Appendix G to further illustrate this point. Note that this task requires the model to meet the demands of multiple properties with precise values, placing high demands on the model's overall understanding capability. Generalpurpose LLMs, or those not specifically trained for this task, lack the required information storage and fitting abilities. The model gain strong molecular understanding capabilities through property enhanced instruction tuning.

4.4 Analyses

Ablation Study. Figure 4 presents an ablation study of the cross-modal matching loss Lmatch and cross-modal contrastive loss Lcontrastivein the PEIT-GEN model for the molecule captioning task (Lst CLM and Lsp CLM are necessary for generation via decoders, thus we do not consider them in ablation study). By removing these training objectives, the performance degradation across all metrics. This demonstrates that both Lmatch and Lcontrastive are helpful in crossmodal feature alignment, thereby enhancing the performance of molecule captioning. We further conducted an ablation study on the above loss components in the molecular property prediction task, as shown in the Table 3 of Appendix G. Impact of SFT steps. Figure 5 and Figure 6 illustrate the outcomes of PEIT-LLM across various tasks with different SFT steps. We observe that the performance consistently improves during the initial epochs for all tasks, indicating that the instructional data is beneficial for each, where the performance tends to plateau around epochs 5-6.

Out-of-Distribution Evaluation. To further evaluate the molecular understanding of PEIT-LLM on unseen data, we tested it on the Mol-Instructions [Fang et al., 2023] test set, without using the full instructions for pre-training and finetuning PEIT-LLMs. As shown in Table 6, PEIT-LLM outperforms all general-purpose LLMs as well as smaller domainspecific models highlighting the strong generalization ability of PEIT-LLM across diverse molecular instruction tasks.

5 Conclusion

We propose PEIT, a framework that enables LLMs to perceive multi-modal features for multi-task molecule generation. PEIT aligns molecular structures, textual descriptions, and biochemical properties through multi-modal representation learning. It leverages templates to synthesize diverse, task-specific instruction data for LLMs. We also introduce a challenging multi-constraint molecule generation task, which requires generating novel molecules that satisfy multiple property constraints. Results show that PEIT outperforms various biomolecular models and LLMs on captioning, generation, and property prediction tasks.

Acknowledgments

The work is supported in part by the National Natural Science Foundation of China (62573372, 62425204, U22A2037, 62450002, 62432011), the Hunan Provincial Key Research and Development Program Project (2025JK2003), the Guangdong Basic and Applied Basic Research Foundation (2026A1515011358), the Shenzhen Natural Science Foundation (JCYJ20250604181610014), and the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (JYB2025XDXM602).