What If LLMs Can Smell: A Prototype

1 Introduction

Large language models (LLMs) are empirically proven to effectively process and fuse multi-modal data, such as images, audio, videos, text and 3D point cloud [Lyu et al., 2023; Wu et al., 2023; Yang et al., 2024; Wang et al., 2024; Han et al., 2023]. They use encoders for different modalities in parallel to encode and align the data and extract semantic understanding by LLM-based models. This kind of multi-modal LLMs is highly limited by the encoders they apply. An additional encoder might need more datasets for modal semantic alignment and far more time-consuming training or fine-tuning. LLMs, on the other hand, can serve as an omniscient Agent by means of generating and using diverse instructions for real-world service APIs and multi-step decision-making processes. For instance, ToolLLM [Qin et al., 2023] facilitates LLMs, e.g., LLaMA, with more than

∗Corresponding authors.

16,000 real-world APIs to enable effective interaction to accomplish complex tasks. It could search reasoning paths and make deliberate decisions to generate generalized answers on an out-of-distribution (OOD) dataset. However, prevalent tool-alike LLMs [Qin et al., 2023; Liu et al., 2024; Gao et al., 2023] do not consider the interaction with multi-modal applications. LLMs for robotics [Vemprala et al., 2024; Brohan et al., 2023] take into account this end, but they hardly talk about the olfaction modality.

Olfactory augmented LLMs can help scent-based descriptions, scent recognition, personalized user experiences and robot interaction. However, few studies are proposed to embody olfaction-related capabilities in LLMs. In [Hassan et al., 2024], the authors provide an effective algorithm which encodes the olfaction and vision sensor data into a multi-modal prompt and instruct the LLM to select a navigation behavior for robots. Sniff AI [Zhong et al., 2024] conducts a comprehensive user study to investigate how well LLMs can interpret human descriptions of scents using an LLM-based guessing model and a scent delivery device. Combining an LLM with a text-image model, OdorAgent [Zhang et al., 2024] can align video-odor semantic understanding and dependently display odor based on images. However, these studies neither de facto embody the smelling capability in LLMs nor consider plugand-play interactive service-oriented e-nose tools.

Inspired by these works, we present an olfactory augmented prototype framework to enable LLMs smelling capabilities. We study liquor classification as an application scenario. Specifically, we customize a dataset w.r.t. three common Korean beers by self-developed four copies of enoses collecting the spatio-temporal data in a distributed manner. We additionally adopt an open-source dataset Liquor2021 [Hou et al., 2022; Meng and Hou, 2021], which provides eight kinds of Chinese liquors on their selfdeveloped e-nose. Six traditional machine learning models [Wijaya et al., 2023] and three deep learning models [Wu et al., 2024; Huang et al., 2017; Karim et al., 2019] are trained on the two datasets and selected based on their performance. Then the trained model is applied in a plug-andplay service API to infer the labels of smelling objects upon the sensor data. A small smelling-related question-answering corpus is generated by ChatGPT 4o mini and filtered by manual labor to fine-tune the selected LLMs (i.e., DeepSeek: R1 Distill Qwen 1.5B and DeepSeek: R1 Distill Qwen 7B [DeepSeek-AI, 2025]). The experiments show that, finetuned within 5 epochs, the LLMs can identify the olfactionrelated question; by two designed instructions, LLMs can generate the API invocation commands and answer the questions based on the requested smelling information. Since the olfaction modality is decoupled from the LLM semantic understanding, the instruction can yet boost LLMs to answer the OOD olfaction-related questions if the application scenario is changed. We ultimately deploy the fine-tuned LLM in a simple web-based system to show its feasibility and deployability (elucidated in the video only).

2 Experiment

This section introduces dataset generation for both the artificial olfaction on Korean beers and olfaction-related question answering, the gas classification models and their experimental results, and the means of fine-tuning LLMs.

2.1 Beer Olfactory Sensor Dataset

We briefly exhibit the design of our self-developed electronic nose and experiments in Fig. 2(a). Totally 19 gas sensors and 22 channels are embedded in each copy. Each of 4 Raspberry Pi 5s functions as an e-nose controller. We additionally developed a simple web-based management system to remotely activate the collection program and collect the sensory data. Since the diffusion patterns of the gases are also significant, we expose our e-noses to air instead of limiting the odors within a traditional pumping chamber. The second figure in Fig. 2(a) shows that the e-noses are located at the four corners of a 0.5m × 0.5m area. The sample container is a transparent acrylic container measuring 15cm × 15cm × 0.5cm. The container is always placed at the exact center of the area. We used five 1L bottles of Terra, Kelly, and Cass (these are the names of the three most popular beers in South Korea) beer each. Each bottle of beer was stored in a household refrigerator at a constant temperature of 5◦C before the experiment. We extracted twelve samples of 50ml each from every bottle to conduct experiments. Each sample was left somewhere for 10 minutes and still at the target position for 1 minute before data collection began. Each experiment lasts for 10 minutes. The sampling rate is nearly 3Hz. All experiments were conducted in a private room with temperature (10∼16◦C) and 20% humidity. We ultimately adopted 5 × 3 × 12 = 180 data samples.

2.2 Gas Classification

We first downsample the Liquor2021 dataset from 1000Hz to 10Hz. Then the Liquor2021 and the beer olfactory sensor dataset are each divided into five parts for 5-fold validation. We adopt six traditional machine learning models (i.e., MLP, k-NN, SVM, Decision Tree, Random Forest, and AdaBoost) [Wijaya et al., 2023] and three deep learning models (i.e., TCN [Wu et al., 2024], DenseNet [Huang et al., 2017] and LSTM-FCN [Karim et al., 2019]) to seek an optimal gas classification model based on their average accuracies. According to our experiment, DenseNet achieved 100% average accuracies on 2 datasets. We thus apply it in its service API to infer the classification results on real-time sensory data.

2.3 Olfaction-Augmented Corpus

We categorize the olfaction-augmented questions into 4 types: closed, funnel, open and meaning-based questions. This categorization is dependent on the question difficulty. To boost the LLMs, we use ChatGPT 4o mini to generate semantically similar questions given manually designed questions. The first part in Fig. 2(b) demonstrates the promptbased question generation. By manually filtering, we got a total of 783 questions (281 closed questions, 51 funnel questions, 299 open questions and 152 meaning-based questions). To form instruction-question-answer triplets, we design two delicate instructions to prompt the LLMs as shown in the second part in Fig. 2(b). For the message in the first round, the LLM identifies if the question is related to the smell; if so, it provides the command to activate the second round 1 . The message in the second round is designed for generating the real answers given the requested gas information. Before fine-tuning, the "⟨ object ⟩" tokens in the questions are substituted by a random label. In addition, we provide dozens of template answers in the second round. We manually split the triplets containing four kinds of questions into the train/valid sets. For the training set, we have four types of questions in the first round but funnel and open questions only in the second round. The same applies to the validation set. We further include all the questions without answers in the test set.

2.4 Olfactory Augmented LLMs

We adopt and fine-tune two DeepSeek models [DeepSeek-AI, 2025]: R1 Distill Qwen 1.5B and R1 Distill Qwen 7B within 5 epochs on the two datasets, respectively. The labels are randomly assigned to questions while training and evaluating. All the fine-tuning experiments share the same group of hyperparameters: 8 samples per training and validation batch, a learning rate of 0.0001, checkpoint saving per 100 steps, and gradient accumulation per 2 steps. The maximum sequence length of LLMs is set to be 384. The experiments were conducted on an HPC (DELL DSS8440) equipped with 8 NVIDIA A100 GPUs at the Biospecimen-Multiomics Digital Bioanalysis Core Facility of Hanyang University. The validation loss curve is shown in Fig. 3.

3 Use Cases

We provide a comprehensive use case study on our selfdeveloped beer olfactory sensor dataset to demonstrate the proposed prototype is effective and feasible. Besides, we do not discuss funnel questions here since they are similar to closed questions. The use cases are exemplified in Fig. 4, where each question is randomly selected from each category.

Closed Questions. As shown in the blue blocks in Fig. 4, both models could answer "Yes/No" precisely to the closed questions; they can provide the correct beer labels. The answers generated are slightly different but semantically similar.

1 We do not generate the instructions alike the others since we only consider real-time application scenarios where only one service API is equipped.

Open Questions. As shown in the green blocks in Fig. 4, it can answer what exactly it is smelling, e.g., "I can tell from the scent that it's Kelly beer". Besides, it additionally provides a short introduction to the beer.

Meaning-based Questions. The answers from the DeepSeek R1 Distill Qwen 7B model provide the exact introduction of the beer without answering "Yes/No". On the contrary, the smaller LLM can answer politely, starting with "yes". It also names the correct label of the beers but loses the exact introduction. We ascribe these limitations to insufficient training and low diversity of the answers. Nevertheless, it indicates that olfactory embodiment in LLMs is feasible, and we are inspired to diversify both the gas dataset and the corpus.

4 Conclusion and FutureWork

This demo presents an olfactory augmented prototype framework for LLMs. The olfaction modality therein is embodied in the LLMs primarily by plug-and-play smelling services and designed instructions. The plug-and-play services are powered by the gas classification models, each trained on one open-source dataset and one customized dataset using self-developed e-noses. Two LLMs are fine-tuned on the self-constructed corpus, which includes four kinds of questions and delicately designed instructions. The use case study and the web demonstration evidence the feasibility and deployability of the proposed prototype.

In the future, we will expand the corpus w.r.t. more odors and collect their sensory data. We will explore more powerful LLMs with retrieval-augmented generation (RAG) to answer questions towards any gas. Further, we consider that this and related applications can inspire many domains, such as virtual reality, robotics or other human-centric products.

Acknowledgments

This work was partly supported by the Foundation of National Key Laboratory of Human Factors Engineering (Grant NO. HFNKL2023J08), the National Natural Science Foundation of China (Grant No. 62441219 and No. 62172252), and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT)(RS-2020-II201373, Artificial Intelligence Graduate School Program (Hanyang University)).