Multimodal Inference with Incremental Tabular Attributes

1 Introduction

Modern datasets often include multiple modalities, with visual and tabular modalities being particularly prevalent in the medical field [Sudlow et al., 2015]. The visual modality is typically derived from imaging techniques such as CT scans, containing rich spatial and anatomical information. In contrast, the tabular modality often records structured patient data, such as medical history, laboratory results, and demographic information, which encapsulates a different set of critical features compared to images [Soenksen et al., 2022;

∗Weimin Tan and Bo Yan are the corresponding authors.

Yi et al., 2023]. Numerous studies have demonstrated the efficacy of integrating visual and tabular modalities, showing that such multimodal approaches often outperform singlemodality models [Huang, 2023; Hager et al., 2023]. However, existing methods generally assume a static tabular modality, requiring the attributes of tabular data at inference time to align with those present during training [Somepalli et al., 2021]. In practical applications, tabular data is frequently updated. For instance, in Alzheimer's disease prediction, existing models have been trained on MRI images with demographic information and clinical indicators. Recently, significant biomarkers such as the YWHAG protein [Guo et al., 2024] and inflammation/immune-related markers [Jack et al., 2024] have been discovered. Leveraging these new biomarkers alongside existing models for retraining poses a challenge, as recalling patients from previous databases for re-evaluation is impractical. Such dynamic nature of tabular modalities underscores the need for models capable of handling incremental tabular attributes in a multimodal context, enabling more robust and adaptable inference in real-world scenarios.

To address this challenge, we introduce a novel task termed multimodal inference with incremental tabular attributes. As shown in Figure 1, we consider a scenario in which a multimodal model has been trained using any deep learning algorithm on visual and tabular modalities. During the inference stage, new attribute columns are introduced to the tabular modality. Our objective is to propose a method that fine-tunes the trained model to leverage the relationships among the incremental tabular attributes, the old tabular attributes, and the visual modality. In doing so, the model learns task-relevant information more effectively in an unsupervised manner. This approach aims to outperform models that discard the new attributes and rely solely on the old ones, while simultaneously reducing the data preparation overhead and improving convergence speed compared to fully supervised re-training.

Designing such a method is challenging, with the main difficulty being the inability to dynamically adapt due to the heterogeneity of the tabular modality. Each column in a table represents a distinct attribute with varying scales, semantics, and statistical distributions [Nam et al., 2023]. Directly applying encoding patterns from old attributes to new ones may not only fail to improve performance but could also degrade it. Furthermore, effectively leveraging the visual and tabular modalities to compress redundant information in incremental attributes and extract novel, task-relevant knowledge is essential. This ensures the new attributes contribute meaningfully to the model. However, research on multimodal tasks involving visual and tabular modalities, particularly when incremental tabular attributes are introduced, remains limited.

To address the aforementioned challenges, we propose a novel framework named Multimodal Inference with Incremental Tabular Attributes (MIITA). Specifically, we design a dedicated encoder for the incremental tabular attributes based on a Variational Autoencoder (VAE). This encoder not only extracts implicit features but also decouples them, ensuring that the learned latent representations are as independent as possible across dimensions. This disentanglement enhances interpretability and makes the latent features more amenable to constraints imposed by the old tabular attributes and the visual modality. Besides classification loss and CLIP loss-encouraging the representations of the two modalities to be close for the same sample and distant for different-we introduce two new loss functions. The first is a disentangled loss designed to further improve the disentanglement of the latent representations with the old representation. The second is a consistency loss, aimed at strengthening the alignment between the incremental tabular attributes and the visual modality. Both losses have demonstrated their effectiveness in ablation, showing significant contributions to overall performance. In summary, the contributions of this paper are:

• This paper introduces a new task, Multimodal Inference with Incremental Tabular Attributes, aiming to enable multimodal models to incorporate incremental tabular columns during the inference stage, enhancing the practicality of AI models in dynamically changed tables.
• To dynamically and unsupervisedly leverage incremental tabular attributes, we propose the MIITA framework, which disentangles the incremental representation from the old ones and adds the modality consistency constraint for better alignment. Our approach achieves state-of-the-art performance across five public datasets.
• Despite the new task, we show that MIITA evokes a new training procedure for general supervised multimodal learning with tabular and vision modalities.

2 RelatedWork

2.1 Multimodal Contrastive Learning

Multimodal contrastive learning has emerged as a powerful framework for learning representations across different modalities. Methods such as CLIP [Radford et al., 2021] and ALIGN [Jia et al., 2021] leverage large-scale paired datasets to align representations of text and images through contrastive objectives. These approaches ensure that similar samples across modalities are brought closer in the shared representation space, while dissimilar samples are pushed apart [Wu et al., 2021]. Recent works have extended this paradigm to other modality pairs, especially visual-tabular data [Hager et al., 2023; Wang et al., 2024a], demonstrating its versatility in fusing visual and tabular information.

However, existing multimodal contrastive learning methods are generally designed for static multimodal datasets and assume fixed input dimensions for each modality. When new attributes are introduced to the tabular modality, these models fail to adapt without retraining on the updated dataset. This limitation hinders their applicability to real-world scenarios.

2.2 Tabular Deep Learning

Tabular deep learning has become an important area of research, focusing on adapting neural networks to structured tabular data. Architectures such as TabNet [Arik and Pfister, 2020], FT-Transformer [Gorishniy et al., 2023] and SCARF [Bahri et al., 2022] have shown promise in feature selection, representation learning, and interpretability for tabular data. These models leverage advanced attention mechanisms or feature gating to capture meaningful relationships between attributes and outperform traditional machine learning approaches in various tasks. Recently, Large Language Models (LLMs) have also been explored for tabular data tasks. Methods like TabLLM [Hegselmann et al., 2023a] and TableGPT2 [Su et al., 2024] convert tabular attributes into natural language formats, enabling LLMs to incorporate prior knowledge and perform reasoning directly on structured data.

Despite these advancements, traditional tabular deep learning models typically assume a fixed tabular input. Current efforts, such as TransTab [Wang and Sun, 2022] and TabPFN [Hollmann et al., 2025], have introduced methods for handling dynamic attributes in tabular data. However, these approaches are predominantly focused on single-modality settings and lack the capability to effectively utilize visual modality information. On the other hand, LLM-assisted approaches excel in zero-shot or few-shot scenarios by leveraging pre-trained knowledge but often fall short in scenarios requiring incremental learning of incremental attributes [Zhang et al., 2024; Carballo et al., 2023]. Consequently, both traditional and LLM-assisted methods face significant challenges in addressing dynamic tabular attributes and fully leveraging cross-modal relationships, leaving a gap in solving incremental multimodal inference tasks.

3 Method

In this section, we formally define the task of Multimodal Inference with Incremental Tabular Attributes (MIITA). We then present the proposed framework, also named MIITA, detailing its architecture and the mechanisms it employs to handle incremental updates in tabular attributes during inference.

3.1 ProblemFormulation

Consider a multimodal dataset D = {(xvis i , xold i , yi)}Ni=1 , where xvis i ∈ X vis represents a visual input (e.g., medical images), xold i ∈ X old denotes tabular data comprising M old attributes, and yi ∈ Y is the target label. Amultimodal model f is trained on D to predict yi by jointly utilizing xvis i and xold i :

where Θ denotes the model parameters learned in training.

In the MIITA setting, during the inference phase, incremental attributes are added to the tabular modality, resulting in an incremental tabular input xinf i ∈ X inf , where X inf ⊃ X old and the incremental attributes are denoted as X inc = X inf \ X old . The goal is to adapt the trained model f to utilize the augmented tabular input xinf i along with xvis i to predict yi:

where f ′ and Θ′ represent the adapted model and updated parameters, respectively.

The MIITA task involves three key challenges:

• Dynamic Adaptation: The model must incorporate X inc without requiring retraining on the entire dataset.
• Preserving Knowledge: The model should retain its ability to utilize the old attributes X old and X vis .
• Unsupervised Adaptation: The adaptation should not rely on labels for the incremental attributes, instead leveraging unsupervised learning to extract task-relevant information.

By addressing these challenges, MIITA enables robust multimodal inference in dynamic real-world scenarios where tabular data evolves over time.

3.2 MIITA Framework

As shown in Figure 2, our framework is based on a trained multimodal model, from which we extract the tabular encoder and the visual encoder, freezing their parameters to ensure that MIITA does not perform worse than the old model. During inference, the image and the old tabular attributes of a given sample are passed through the visual encoder and tabular encoder, respectively, producing the visual representation and old tabular representation.

To handle incremental tabular attributes, we use a dedicated Disentangled Representation Learning (DRL) encoder. This encoder disentangles the entangled representations of the incremental attributes into interpretable dimensions, such as spatial or temporal factors. This disentanglement not only enhances the interpretability of the model but also strengthens the effectiveness of subsequent multimodal learning procedures. After passing through the DRL encoder, we obtain the incremental tabular representation.

Our framework leverages the relationships among the three representations: visual, old tabular representation, and incremental tabular representation, using a combination of novel loss functions. Beyond the foundational InfoNCE loss and a task-specific classification loss with pseudo-labels, we will introduce novel Incremental Disentangled Loss and Visual Consistency Loss in detail. By integrating the DRL encoder with these four loss functions, we finally use the improved inference tabular representation for downstream tasks. Our framework can effectively utilize the information from the incremental tabular attributes, improving model performance in an unsupervised manner while achieving faster convergence.

DRL Encoder

The DRL encoder is pivotal for disentangling incremental tabular attributes, enabling effective utilization of new information in the MIITA framework. Specifically, it employs a β-Variational Autoencoder (β-VAE) to encode the incremental attributes xinc into a disentangled latent space zinc , ensuring independence among dimensions [Higgins et al., 2017]. Other DRL encoder variants, like FactorVAE [Kim and Mnih, 2019] and DIP-VAE [Kumar et al., 2018], are also compatible with our framework. The β-VAE's objective is defined by a combination of reconstruction and regularization losses.

The reconstruction loss ensures that the encoder effectively captures the information contained in xinc :

where ˆxinc is the reconstruction of the input. To promote disentanglement, a regularization term enforces the independence of latent dimensions by introducing a KL divergence between the posterior qϕ(zinc |xinc ) and a factorized unit Gaussian prior p(zinc ):

The overall loss for the DRL encoder is expressed as:

where β is a hyperparameter that controls the trade-off between reconstruction accuracy and disentanglement strength.

Incremental Disentangled Loss

In addition to disentangling the incremental attributes themselves, we aim to ensure that zinc encodes meaningful, complementary information while avoiding redundancy with zold . To achieve this, we introduce a disentangled loss with two distinct focuses:

The redundancy mitigation loss focuses on maximizing the KL divergence between qϕ(zinc |xinc ) and q(zold ), ensuring zinc captures novel, nonlinear information not present in zold :

The covariance regularization loss, on the other hand, penalizes linear cross-correlations between the dimensions of zinc and zold , further promoting disentanglement by encouraging independence:

where C cross ij is the cross-covariance matrix in zinc and zold .

The overall disentangled loss is then defined as:

where λ1 and λ2 are hyperparameters controlling the tradeoff between nonlinear redundancy mitigation and linear covariance regularization.

InfoNCE Loss

The InfoNCE loss [van den Oord et al., 2019] facilitates alignment between the visual and tabular modalities by encouraging the inference tabular representation zinf , formed by concatenating zold and zinc , to be close to the visual representation zvis of the same sample while being far from those of other samples in the batch.

The inference tabular representation is defined as:

The InfoNCE loss is expressed as:

where sim(a, b) denotes cosine similarity, τ is a temperature hyperparameter, and B is the batch size. This loss ensures that zinf aligns closely with zvis for the same sample while preserving inter-sample separability.

Visual Consistency Loss

To better align the tabular and visual modalities, we further ensure that the inference tabular representation zinf is closer to the visual representation zvis than the old tabular representation zold alone. This encourages incremental tabular attributes to complement the old tabular representation effectively.

To enforce the desired alignment, we propose an InfoNCEinspired loss that compares the relative similarities between the representations:

By minimizing LVis-Cons, the model explicitly enforces the inference tabular representation zinf to capture richer and more complementary information from the incremental attributes, improving its alignment with the visual modality. This approach leverages the additional information from incremental tabular attributes to enhance multimodal inference while maintaining consistency with the visual modality.

Classification with Pseudo Labels

To enhance the framework's classification capabilities, pseudo-labels generated by old models are employed to provide supervision in an unsupervised setting. To ensure reliability, only pseudo-labels with a confidence score higher than a specific threshold (e.g., 0.7) are retained for supervision. The classification loss is defined using the standard cross-entropy loss as [Mao et al., 2023]:

where ˆ yi represents the highest pseudo-label probability of sample i, yi is the predicted probability for the corresponding class.The purpose of LCls is to ensure that the learned features are strongly related to downstream tasks by aligning them with task-relevant information. This process helps to discard task-irrelevant features, thereby improving the effectiveness of MIITA for specific applications.

Overall Objective

The overall objective integrates all components to optimize the MIITA framework holistically:

where each term contributes to specific aspects of representation learning, alignment, and classification. By balancing these losses, the model effectively captures and integrates multimodal information, achieving robust and interpretable performance.

4 Experiment

4.1 Experiment Setup

Datasets and Comparative Methods

We use five public datasets that are commonly used in multimodal learning areas using tabular and visual modalities: ADNI (AD) [Jr et al., 2008], Data Visual Marketing (DV) [Huang et al., 2023], Pokemon Primary Type (PK), HearthStone Card's category (HS), CS:GO skin quality (CG) [Lu et al., 2023]. In each, we remove specific columns from the train and validation sets while keeping them in the inference set to simulate real-world incremental tabular inference. The deleted columns were chosen logically, reflecting their historical appearance. Details are in the appendix.

For baselines, we use single tabular modality classifiers: XGBoost [Chen and Guestrin, 2016], SCARF [Bahri et al., 2021] and FT-Transformer [Gorishniy et al., 2021]. Since these tabular methods can't utilize incremental columns, we only use the same columns as the train set for inference. Single visual modality classifiers like CNN [O'Shea and Nash, 2015] and ViT [Dosovitskiy et al., 2021] are also used. Then, we compare our method to recent tabular-visual multimodal models: AutoMM [Shi et al., 2021], SimCLR [Chen et al., 2020], MMCL [Huang, 2023] and TIP [Du et al., 2024]. Also, we compare our method to FT-Trans* and TransTab [Wang and Sun, 2022], which can utilize all columns in the inference set. Note that in our experiments, we modified FTTransformer by replacing the embedding layer with a linear embedding (1 × encoder size) to handle any input dimensions, noted as FT-Trans*. Finally, we compare performance with LLM-assisted models, including TabLLM [Hegselmann et al., 2023b] and MediTab [Wang et al., 2024b].

Implementation Details

We encode categorical features using a backward difference encoder [Potdar et al., 2017]. Key setting of MIITA are: β (VAE) ∈ {2, 5, 10}, λ1 (KL) / λ2 (cov-cross) ∈ {0.5, 1.0, 2.0}, and pseudo-label threshold = 0.7. The loss weights in Eq.13 were adjusted based on early gradient magnitudes. The default setting is {1, 0.5, 0.8, 0.4, 1}.

4.2 Results on Public Datasets

We evaluated the MIITA framework for six tasks in five widely used datasets that span entirely different domains, each containing both tabular and visual modalities. As shown in Table 1, MIITA consistently outperformed state-of-the-art (SOTA) single-modal and multimodal methods, representing MIITA is a general framework applicable for all scenarios including medical, advertisement and game. The visualization result of two difficult samples is shown in Figure 3. All results were averaged over four runs to mitigate randomness, with the corresponding variances provided in the appendix.

Single-Modal Baselines

We first conducted experiments using single-modal models for both tabular and visual data. Since traditional tabular learning methods cannot handle dynamic attributes, our tabular experiments only used the static, old attributes matching the training setup. The results demonstrate significant performance limitations in both modalities: the visual data proved unstable and informationally limited, while the tabular modality was severely constrained by its inability to utilize incremental columns.

Multimodal Methods

Three multimodal learning models designed for tabular and visual modalities were evaluated. These models achieved superior performance compared to single-modal baselines, highlighting the effectiveness of multimodal learning in leveraging complementary information from both modalities. We used SimCLR as a baseline for further experiments, replacing its tabular feature extractor with methods capable of handling dynamic attributes.

Experiments with Incremental Tabular Attributes

To explore handling dynamic tabular attributes, we first tested FT-Trans*, which directly maps tabular data with incremental attributes using the old encoder without adaptation. The results were poor, demonstrating the necessity for tailored adjustments in encoding schemes. Both the traditional selfsupervised pretraining approach (TransTab, TIP) and large language model (LLM)-assisted tabular learning also failed to achieve satisfactory results. Their performance was often worse than multimodal models using only the old tabular attributes because they cannot put trained tabular encoders into fully usage. In contrast, MIITA consistently achieved leading performance across all tasks. This demonstrates the superiority and generalizability of the MIITA framework in addressing the challenges posed by incremental tabular attributes in multimodal learning.

4.3 Generalizability Across Different Scenarios

As discussed in the methodology section, MIITA is designed to be a model-agnostic framework, capable of integrating with various architectures seamlessly. As illustrated in Figure 4, MIITA demonstrates consistent performance improvements when applied to models employing different tabular and visual feature extractors in AD3 dataset. This showcases the flexibility of MIITA in adapting to diverse base architectures without requiring significant structural modifications.

The generalizability of MIITA extends beyond tabularvisual multimodal tasks to scenarios involving other modalities. We conducted experiments on additional multimodal learning tasks: tabular-text in a game dataset[Lu et al., 2023] and tabular-temporal data in a stock dataset. Across these tasks, MIITA achieved a consistent performance improvement, highlighting its effectiveness in leveraging incremental tabular attributes with other modalities, reinforcing its potential as a universal approach for dynamic multimodal learning.

4.4 General Improvement on Supervised Learning

Beyond excelling in the proposed MIITA task, our framework also demonstrated significant advantages in fully supervised multimodal scenarios. The experiments showed that MIITA consistently outperformed traditional training approaches when labeled data was available.

In the supervised setting, the MIITA framework was applied by first disentangling the complete tabular data using the DRL encoder. The disentangled features were then treated as original inputs and progressively grouped in batches to train the MIITA framework with the proposed loss constraints. For this scenario, the DRL encoder utilized a Transformerbased architecture. As presented in Table 2, this incremental training approach achieved superior performance across three datasets compared to state-of-the-art multimodal learning models [Hager et al., 2023]. Furthermore, we evaluated the disentanglement level using the metric introduced in [Higgins et al., 2017], demonstrating that MIITA achieves a higher disentanglement rate compared to traditional multimodal methods, which is beneficial for the integration of multimodal learning. These findings highlight the broader applicability of the MIITA framework and its potential to inspire novel strategies for progressive training and disentangling in general tabular-visual multimodal learning tasks, moving beyond the limitations of holistic learning approaches.

4.5 Incremental Tabular Attributes Settings

As shown in Table 3, we explored the impact of different incremental attribute settings in the tabular modality on the performance of MIITA. Based on human cognitive perception, we categorized the tabular attributes into three groups: those highly related to the image (i.e., the table attributes are fully reflected in the image), moderately related, and weakly related. Experimental results demonstrate that weakly related tabular attributes provide more information for multimodal tasks and play a more significant role in incremental inference. We also tested different numbers of incremental attributes, categorized as few, moderate, and many. Results indicate that as more attributes are added, MIITA's performance generally will improve. Even with a small incremental column size, MIITA shows performance gains, meaning MIITA treats incremental attributes as valuable rather than noise.

4.6 Ablation Studies

To validate the effectiveness of the MIITA framework and the proposed loss functions, we conducted ablation studies by removing or replacing key components, as summarized in Table 4. Removing the visual consistency loss caused a notable performance drop, emphasizing its role in aligning the tabular representation with the visual modality. Replacing the disentangled encoder and incremental disentangled loss with a standard transformer also led to reduced performance, highlighting the benefits of disentangling incremental tabular attributes into interpretable dimensions. Additionally, using the disentangled encoder without the incremental disentangled loss caused a significant performance decline, showing that isolating incremental attributes leads to redundancy and inefficiency. The full MIITA framework, with all modules included, achieved the best results, demonstrating the effectiveness of each component in addressing the challenges of incremental tabular attributes in multimodal learning.

5 Discussion and Conclusion

In this work, we introduced MIITA, a novel framework designed to address the challenge of multimodal inference with incremental tabular attributes. Our extensive experiments demonstrated that MIITA outperforms state-of-the-art singlemodal and multimodal methods across a variety of datasets. The key strength of MIITA lies in its ability to effectively integrate incremental tabular attributes, leveraging disentangled representation learning and innovative loss functions to maintain alignment with existing modalities. We believe that MIITA offers broad applicability to multimodal learning tasks involving tabular data, and its design principles can be extended to other domains like audio and video. MIITA represents a significant step forward in integrating dynamic tabular data with other modalities and opens new opportunities for future research in flexible, interpretable multimodal learning.

Acknowledgements

This work is supported by NSFC (grant nos. 62472102 to B.Y. and 62372117 to W.T.) and by the Natural Science Foundation of Shanghai (24ZR1490400 to W.T.). The computations in this research were performed using the CFFF platform of Fudan University.