Empowering Multimodal Road Traffic Profiling with Vision Language Models and Frequency Spectrum Fusion

1 Introduction

The rapid development of smart cities facilitates the digitalization of road traffic applications, i.e., the importance of understanding and accurately analyzing road traffic conditions cannot be overstated [Xu et al., 2023b]. Efficient transportation management and profiling are essential for the smooth functioning of economies, ensuring the timely delivery of goods and services, and facilitating the daily commute of people [Chen et al., 2023; Fan et al., 2022]. Urban planning also heavily relies on a comprehensive understanding of traffic conditions to design sustainable and livable cities [Yan et al., 2024]. Moreover, public safety is closely intertwined with traffic prediction, as accidents and congestion can pose significant risks to the well-being of individuals. Thus, road traffic profiling equips decision-makers with essential insights by means of traffic prediction, transportation management, and classification of road traffic conditions [Liu et al., 2024].

Traditional methods of road traffic profiling apply basic machine-learning techniques to analyze road traffic conditions, which fails to handle complex and dynamic modern traffic systems [Sommer et al., 2010]. Besides, web-sourced data with large volumes presents greater challenges to the accuracy of these traditional methods. Web sources can provide multimodal information, including real-time traffic updates, images captured by satellite or surveillance cameras, social media posts related to traffic incidents, and textual descriptions of traffic conditions [Xu et al., 2023a]. How to effectively utilize these multimodal data for road traffic profiling is highly challenging.

According to task-specific learning, we can divide the road traffic profiling problems into four categories: traffic flow analysis, accident detection and analysis, fire detection and analysis, and travel time estimation [Liu et al., 2023]. Deep learning techniques, like convolutional neural networks based methods [He et al., 2016; Alam et al., 2023], recurrent neural networks based methods [Jin et al., 2017; Zheng et al., 2020], graph neural networks based methods [Zhang et al., 2023; Deng et al., 2024], and transformer-based methods [Lin et al., 2022a; Xu et al., 2024], have been widely used to analyze these traffic conditions, which helps to accurately identify images or text descriptions of road traffic and make driving decisions. However, as the current road traffic scenario is complex and ever-changing with multimodal data sources, merely analyzing the information in images or text is insufficient to capture the comprehensive road traffic conditions, thereby affecting traffic decision-making and travel safety.

Large language models (LLMs) possess a strong capacity for handling multimodal data, particularly in the understanding and generalization of textual data [Khattar et al., 2019; Feng et al., 2024], and offer more comprehensive information understanding via the complementarity of modalities to address information loss [Chen et al., 2022]. As an example, Qian et al. [Qian et al., 2021] proposed a multimodal framework that jointly captures multimodal context information and the hierarchical semantics of text by BERT and ResNet, which enhances detection accuracy by fusing inter-modality and intra-modality relationships. To optimize multi-vehicle dispatching and navigation in smart cities, Chen et al.[Chen et al., 2024] proposed an LLM-driven framework for efficient task allocation and an RL-based module for cooperative navigation and handling heterogeneous vehicles. Recently, Yan et al. [Yan et al., 2024] utilized LLMs to enhance the textual information and adopted contrastive learning for image and text fusion, which produces multimodal representations for urban region profiling. These methods analyze multimodal data by using the powerful information extraction ability of LLMs, but the task-specific applications hinder the exploration of road traffic profiling. Currently, it is still a challenging task to analyze complex road traffic conditions with web-sourced images and a powerful complement of textual data.

To address the above issues, we propose the first-ever multimodal fusion framework (TraffiCFUS) in road traffic profiling, which refines the text with the prompt design of Vision Language Models (VLMs) and optimizes the multimodal representations with frequency spectrum learning. The main contributions of our work are three-fold:

• TraffiCFUS is a very early attempt for multimodal learning and fusion framework for road traffic profiling, which introduces VLMs to generate text and then creates tailored prompt instructions for refining this text.
• We convert the multimodal data from the spatial domain to frequency domain and perform a cross-modal spectrum transform to filter out irrelevant information. Four losses with different aspects are designed to construct the objective loss for downstream task optimization.
• Extensive experiments on four public traffic datasets illustrate the effectiveness of our framework. We also conduct a series of ablation studies to show the influence of different components in our framework. Our source code is available at https://anonymous.4open.science/r/ TraffiCFUS-87EF.

2 RelatedWork

2.1 Deep Learning for Road Traffic Profiling

Deep learning techniques have been widely applied in traffic and driver profiling, demonstrating significant advancements in understanding and predicting complex behaviors within these domains [Dui et al., 2024]. For example, Cura et al. [Cura et al., 2020] utilized data from the CAN Bus system to train LSTM and 1D-CNN models, finding that CNNs are particularly effective at distinguishing aggressive driving styles. This research underscores the importance of deep learning in driver profiling, with applications extending to fields like insurance and fleet management. Additionally, Abdelrahman et al. [Abdelrahman et al., 2020] presented a robust datadriven framework leveraging supervised machine learning to evaluate drivers' risk profiles. Using the SHRP2 dataset, they identified key risk factors through Random Forest and Deep Neural Network models, enabling real-time risk assessment via cloud-based applications. Sekula et al. [Sekuła et al., 2018] introduced a machine-learning framework trained on vehicle probe data for traffic volume estimation. Conducted in Maryland, this study showed that neural networks integrated with profiling methods can improve volume estimations by 24% over traditional techniques. These approaches demonstrate the power of deep learning models in addressing the challenges of road traffic profiling, but they fail to utilize multimodal data of text and image simultaneously to train the real-world applicable models.

2.2 LLMs for Road Traffic Profiling

Recent advances in traffic analysis have leveraged Large language models (LLMs) for road traffic profiling [Cao et al., 2024], particularly in traffic condition classification where LLMs, such as GPT-2 and GPT-3, are exploited. Chen et al. [Chen et al., 2024] proposed LiMeDa, which combined an LLM for efficient task allocation and an RL-based module for cooperative navigation and handling heterogeneous vehicles. However, LiMeDa has not fully addressed the extraction and integration of multi-modal information, which is crucial for effectively handling complex tasks across both visual and textual data. Hu et al. [Hu et al., 2024] extended a multimodal LLM (BLIVA) that combined visual and textual information, by leveraging learned query embedding and encoded patch embedding, to address complex visual questionanswering (VQA) tasks. Yang et al. [Yang et al., 2024] introduced EMMA, which trained VLM agents through crossmodal imitation learning to adapt better to visual world dynamics. Besides, Li et al. [Li et al., 2024] designed OmniActions, which introduced a novel pipeline to process multimodal sensory inputs and form explicit reasoning. OmniActions made it possible to predict context-aware digital actions based on an integrated design space.

However, these approaches did not fully combine the multimodal data to generate powerful representations for road traffic profiling. In this paper, we aim to leverage LLMs to fuse multimodal data, which enables a more comprehensive understanding of road traffic conditions and provides a multimodal framework for accurate road traffic profiling.

3 ProblemDefinition

Definition 1 (Traffic Area). According to the previous studies, we can divide the whole transportation of a city into M traffic scene areas for road traffic profiling.

Definition 2 (Traffic Image). By using cameras or other sensor devices, traffic scene images are captured to depict various aspects of traffic conditions in a traffic area A, including images of vehicles, pedestrians, traffic signs, signals, and the overall road/weather environment. Each input traffic image can be represented as gA, which satisfies the condition:

gA ∈ RL·H·3 , L and H are length and width.

Definition 3 (Traffic Description). We use text description tA to show the traffic condition in a traffic scene area A. These text descriptions can be extracted from traffic scene images by image captioning tools, i.e., the well-trained VLMmodels, such as VL-Plus, Llama-adaptor and InternVL2.

Definition 4 (Traffic Category). Different traffic scene Categories represent different traffic conditions. Each category provides distinct insights into the specific characteristics and challenges of that particular traffic condition, which is crucial for effective traffic profiling. In this paper, we use four categories Cat = {Cf ,Ca,Cc,Cl} to represent fire, accident, congestion and light traffic, respectively.

The processing of traffic conditions is defined as a multimodal classification problem, where multimodal u contains two parts of image g and text t. Given a traffic dataset D = {d1, d2, ..., dn}, each data instance is denoted as (xi, yi) and xi = {xg , xt }, where xg represents the image element and xt represents the text element. Besides, yi = {1, 2, ...,m}(m ≥ 2) and m represent the number of types of traffic conditions, i.e., yi is the label of data instances. When m= 2, the traffic scene prediction is a binary classification problem, otherwise, it is a multi-class classification problem. Then, a mapping function F is designed to map all the data, including traffic image, text and other environmental data (e.g., road, pedestrian), to a vector vA = F(gA, tA). Finally, this vector can be used to predict the traffic condition by vA → yi.

4 Methodology

We propose a text-refined road traffic profiling framework (TraffiCFUS) with multimodal interaction and fusion, which can effectively identify multimodal traffic conditions from the web. As shown in Figure 1, our method consists of four parts: text refinement and feature extraction, uni-modal spectrum compression, cross-modal spectrum transform, and contrastive learning. The first step is the feature extraction of images and text, which applies VLMs to refine text and generate the initial representations. Then, these features are transformed into spectral features by discrete Fourier transform (DFT). The second step is to compress the uni-modal spectrum and use a filter bank to filter out useless information. The third step is enhancing the informative spectrum and suppressing the irrelevant spectrum by the cross-modal transform. Finally, the spectral features are converted back into the spatial domain by inverse discrete Fourier transform (IDFT) and the output features are used for loss calculation with contrastive learning.

4.1 Text Refinement and Feature Extraction

The first step of TraffiCFUS is to generate texts from images and enhance these texts by prompt design with VLMs. As shown in Figure 2, an image-to-text foundation model, Qwen-VL plus version [Yue et al., 2024], is applied to analyze images and generate descriptive text. With its ability to accurately describe images, Qwen-VL can enhance the way we interact with traffic scenes and improve the accessibility of information. But the text directly generated by VLMs is rough and contains a lot of useless information, which will affect the analysis and classification of traffic conditions. As demonstrated in Figure 2, an example is presented to illustrate that the descriptions of road traffic conditions are highly redundant and contain irrelevant events.

The process for the above text is to use the Qwen-VL and design an instruction-based prompt to optimize the generated text. We have designed three instructions to optimize the description of traffic conditions. One is to get a detailed description of all traffic conditions, including vehicles, pedestrians, traffic signs, signals, etc. Another is to remove redundant or non-traffic content. The other is to simplify the text description and retain the most crucial information. As a result, the text is enhanced with a concise and high-quality summary for traffic condition judgment, shown in Figure 2.

Given a traffic area A with its satellite image gA, we divide it into a sequence of patches gp with dimensions h×w (height × width). Then, a patch embedding is used to embed the image into a dense vector as:

where Wp and bp are random variables, which need to be trained by models. The final embedding of a image consists of a patch embedding and a positional embedding egpo . Then, the final embedding of an image can be calculated by eg′ p = egp + egpo . To produce a meaningful representation, a pre-trained CNN, such as VGGNet, ResNet, or Inception, is typically used to calculate the patch features:

An image consists of several patches with the features represented as xg = {P t1, P t2, ..., P tn}.

For textual feature extraction, LLMs like BERT (Bidirectional Encoder Representations from Transformers) is used to generate a latent textual representation. BERT takes a sequence of words as input and outputs a context-aware embedding for each word. To obtain a text embedding from BERT, one can take the average or sum of the output embeddings for all the words in the text, or use the embedding of a special token (like [CLS]) that represents the entire text. Given a text sequence xt = {Se1,Se2, ..., Sem} (m is the number of words), we can obtain the word embedding etw and the positional embedding etpo to encode each word. Then, the feature representation of one word can be calculated by:

A text consists of a sequence of words that has m feature representations.

4.2 Uni-modal SpectrumCompression

The spectrum of an image can reveal the distribution of different frequency spectrum components in the traffic image. The low-frequency part usually corresponds to the overall outline of the image and slowly changing areas, such as the background of a large area of traffic roads, vehicle conditions, pedestrian conditions, etc. The high-frequency part represents the details, edges, and texture information in the image. By analyzing the spectrum, features within a specific frequency range can be extracted in a targeted manner and used for tasks such as the analysis of traffic images and the classification of target categories. As shown in Figure 1, Discrete Fourier transform (DFT) is applied to transform the spatial features into spectrum features. Given the spatial signal xg , the Fourier transform of the image embedding is defined as:

where Xg ∈ Cn×d is a complex tensor, k is the frequency index, and j is the imaginary unit. Thus, we can get the image spectrumXg [k] at the frequency 2πk n .

Besides, we can obtain the Fourier transform of the text embedding in a similar way, defined as:

where Xt ∈ Cm×d is a complex tensor. Different text in an image has distinct spectral characteristics. For example, smooth text has less high-frequency content than rough text. Analyzing the above spectrum can help in road traffic profiling and image classification.

Then, we use uni-modal spectrum compression to process spectral characteristics, achieving a greater degree of feature compression while retaining more important information. We introduce a filter bank for each modality Xa (a ∈ {g, t}), which can divide the input frequency spectrum into multiple sub-bands. For uni-modal spectrum compression, the filter bank can be designed to target the specific frequency range where the uni-modal spectrum is located. For an FIR filter bank, the impulse response Ha = [ha1 , ha2 , ..., hak ] of k filter can be designed using windowing techniques. For example, we can use a Hamming window to obtain the following filter:

where w[i] is the Hamming window function and hideala is the ideal impulse response for the filter. The Hamming window function is given by: w[i] = 0.54 - 0.46cos( 2πi k-1 ), and k is the length of the filter. Through the uni-modal spectrum compression, we can get the spectrum as:

where a ∈ {g, t}, l represents the length of each modality, and ⊙ represents the element-wise multiplication. the design of |Xa |2 smooths the spectrum and highlights the main components, which helps with the subsequent learning of unimodal compression. Besides, the Hamming window function helps to aggregate the main information in the traffic features, realizing efficient frequency domain feature compression.

4.3 Cross-modal SpectrumTransform

Fixed spectrum compression has limited contributions in processing traffic images and cannot remove the higher noise in the frequency spectrum. Therefore, we introduce the average pooling scheme to enhance informative components. Average pooling smoothes the spectrum by reducing high-frequency noise and random fluctuations. By preserving overall trends and making the spectrum more regular, it enhances the efficiency of compression algorithms and identifies traffic image features more effectively.

Then, we also make a cross-modal spectrum transform for the uni-modal spectrum of image and text. This process offers an enhanced representation by capturing more complex relationships and features, improves understanding of the connections between visual and semantic aspects, enables multimodal analysis for complex road traffic, and provides robustness by being less affected by noise and variations in either modality. The enhanced spectrum Xa e with cross-modal transform can be calculated by:

where ⊙ represents the element-wise multiplication, Ave(·) represents the average pooling function, and Φa′ represents a matrix that has the same dimension with Xa′ .

After obtaining the enhanced frequency spectrum, we employ the inverse discrete Fourier transform (IDFT) to convert them back to the spatial representations for loss construction. The calculation of IDFT in image and text is:

Through the processing of the DFT and cross-model transform, we can fuse the multimodal features of images and texts and filter out unimportant information, thus providing more reliable features for road traffic profiling.

4.4 Multimodal Interaction and Fusion with Contrastive Learning

Contrastive Loss. To train the parameters of the proposed multimodal method, we apply a dual contrastive learning scheme. The first is a supervised loss Lsup that optimizes the model by maximizing the similarity of samples within the same class and minimizing the similarity of samples from different classes. Given a mini-batchM that contains |M| data instances, we can divide samples into l types {L1,L2, ...Ll} according to the traffic types of datasets. For an instance sj ∈ Li, we can produce a pairwise (sj, sp) to calculate the loss, where sp ∈ Li, p ̸= j. Then, we can define the pairwise objective function with each instance and different types of samples Li(xa , xa ), a ∈ (g, t) [Lin et al., 2022b]. The whole supervised contrastive learning loss is calculated by:

where N = {L1,L2, ...Ll}, |Li| represents the number of instances in Li.

The second is an unsupervised loss that optimizes the model by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. Specifically, InfoNCE loss [He et al., 2020] is applied in our scheme by a pairwise contrastive loss Ls(xg , xt ) and Ls(xt , xg ). The whole unsupervised contrastive loss consisted of two InfoNCE losses, defined as follows:

where |M| represents the number of instances in a selected batch.

Multimodal Interaction Loss. Previous multimodal interaction is very shallow and fails to learn the multimodal representations well. Thus, we design a multimodal interaction loss based on the work [Yu et al., 2020] and a multimodal fusion loss with distribution similarity learning. The decoder architecture based on the Transformer is utilized to merge unimodal visual and textual representations into multimodal ones. Specifically, the multimodal cross-attention uses image modality as the query and text modality as the keys and values. After the interaction, we can generate the textual description for the road traffic profiling by optimising the language modelling loss Lmim. This interaction loss can be learned through minimizing the conditional likelihood of the text description tA in traffic area A, which can be calculated by:

Multimodal Fusion Loss. Except for the multimodal interaction, we also design a multimodal fusion scheme by measuring the distribution similarity between image and text representations. Specifically, Jensen-Shannon (JS) divergence between two types of representations is measured to calculate the distribution similarity and then outputs the classification of traffic conditions. To generate the posterior probability of the training instances, we need to produce an approximation of sample distribution I. Then the posterior probability of the image and text can be represented as I(ug |xg ) and I(ut |xt ), separately. The JS divergence of fusion modal in xa can be calculated as:

where ζ represents the similarity measured by JS divergence, and JS(·) represents the JS divergence function. Then, we can use the distribution similarity to calculate the representations after multimodal fusion as follows:

whereW g andW t represents the training parameters of image and text. Through a fully connected layer, we can obtain the labels of data instances ˆ y.

In road traffic profiling, the classification of traffic conditions is an important application, which helps to predict future congestion on roads and ensure safe travel. Because traffic classification is a multi-classification problem, we apply the multi-class cross-entropy loss as the final fusion loss:

where l represents the number of labels of traffic conditions.

Finally, the objective loss can be calculated by two contrastive losses, multimodal interaction loss, and multimodal fusion Loss, which are denoted as:

where α, β, and γ are three hyperparameters to balance the influence of different losses.

5 Experiments

Comprehensive experiments are conducted on four realworld road traffic datasets to address the research questions (RQ) outlined below.

• RQ1: Does TraffiCFUS outperform existing approaches in road traffic classification tasks?
• RQ2: What is the impact of different components (e.g., enhanced text, multimodal interaction) in TraffiCFUS?
• RQ3: How effective is the feature visualization of the proposed TraffiCFUS framework?

5.1 Experimental Setting

Baselines. Our framework is compared with eight state-ofthe-art baselines: Resnet-18 [He et al., 2016], att-RNN [Jin et al., 2017], EANN [Wang et al., 2018], MVAE [Khattar et al., 2019], HMCAN [Qian et al., 2021], CAFE [Chen et al., 2022], LogicDM [Liu et al., 2023], and UbanCLIP [Yan et al., 2024]. We evaluate all methods on four realworld datasets, including "Traffic-Net 2", "Traffic-Net 4", "DAWN", and "TCN". We refer the reader to Appendix 1 for more details of the baselines and datasets.

Metrics. The performance is evaluated by Accuracy, Precision, Recall, and F1 score metrics[Liu et al., 2023; Yan et al., 2024]. To ensure a fair comparison, we follow the optimal parameter settings of the baselines. Please see the Appendix for more details about the parameter settings. All experiments are run 15 times and averaged results are reported.

1 Please refer to the version of this paper with Appendix in arXiv.

5.2 RQ1: Performance Comparison

We conduct comparative experiments on four datasets of road traffic classification, which helps to evaluate the accuracy performance of TraffiCFUS and the baseline methods. Table 1 shows the experimental results for the binary classification tasks in sparse vs. accident and sparse vs. fire. Our framework outperforms other baselines on almost all metrics, except for the results of recall and precision on the task "Sparse and Fire". But the f1 score of our framework ranks the top one on this task. Besides, LogicDM performs well on task "Sparse and Accident". HMCAN performs well on task "Sparse and Fire". Although UrbanCLIP ranks the first result on the recall and precision of the task "Sparse and Fire", this method's performance is less stable across different datasets. The classical methods, like att-RNN, EANN and MAVE did not work well on both tasks.

Table 2 details the experimental results of complex realworld datasets on different methods. We can conclude that TraffiCFUS performs the best on all datasets compared with other baselines. On dataset "TrafficNet 4", the following model is HMCAN, which is about 2% worse than our proposed framework. The second ranking method UrbanCLIP is about 3% worse than our proposed framework on dataset "DAWN" and LogicDM is about 2% worse than our proposed framework on dataset "TCN". Besides, Resnet-18 and att-RNN have poor performance on multi-class classification, indicating the limitations in multimodal data processing. The above experimental results and analysis confirm the superiority of our framework, with a detailed feasibility analysis provided in the Appendix.

5.3 RQ2: Ablation Studies

To investigate the influence of different modules in the proposed framework, we conduct a series of ablation experiments on dataset "TrafficNet 2". Specifically, we remove the text refinement module (denoted as -w/o TR), the multimodal interaction module (denoted as -w/o MIM), the fusion module (denoted as -w/o FUS), and the contrastive loss (denoted as -w/o CL) from the TraffiCFUS framework to show the performance on accuracy and F1 score. The detailed experimental results of these components' influence are shown in Table 3, which illustrates that the lack of any component will reduce the accuracy performance of our proposed framework.

Firstly, the absence of text refinement results in the generation of text containing irrelevant information and significant redundancy, which interferes with information extraction and leads to lower model accuracy. Without the multimodal interaction module, the multimodal interaction loss can not be obtained, leading to insufficient integration of visual and textual representations and inadequate multimodal representation learning. The absence of the fusion module meant that the model directly uses the inverse Fourier-transformed text and image features for prediction, hindering effective multimodal information learning. Without the contrastive loss module, the model's performance in classification accuracy, feature representation separation, and multimodal feature fusion declines, which impairs the overall effectiveness of traffic profiling. In summary, the model achieves optimal performance only when all these modules are included.

5.4 RQ3: Feature Visualization

To further investigate the impact of various modules on the features generated by the model, we utilize T-SNE [van der Maaten and Hinton, 2008] dimensionality reduction to visualize the features before they enter the final linear layer, as shown in Figure 3. The dataset used for this analysis is the sparse traffic & accident sub-dataset. In the visualization, green points represent samples with sparse traffic, while grey points denote samples with accidents.

The visualization reveals that TraffiCFUS and its various ablated versions exhibit different degrees of overlap in feature representation. Notably, the full TraffiCFUS model shows the clearest boundaries between the two classes, indicating more distinct and well-separated feature representations. The ablated models (-w/o TR, -w/o MIM, -w/o FUS, and -w/o CL) demonstrate varying levels of feature overlap, which correlates with their reduced performance in classification tasks.

6 Conclusion and FutureWork

This paper investigates different road traffic scenarios, including sparse traffic, congested traffic, accidents, fires, and weather conditions, to make a full profiling of traffic conditions. To analyze the multimodal data on traffic, we propose a first-ever multimodal fusion framework (TraffiCFUS) for road traffic profiling. Powered by VLMs, TraffiCFUS generates enhanced textual information from web-sourced images to assist in road traffic profiling. Moreover, TraffiCFUS filters the irrelevant information and keeps the informative representations by converting the spatial features to spectrum features. The design of multimodal feature interaction and fusion further enhances the extracted representations. Finally, extensive experiments demonstrate the superiority of our proposed framework. Ablation studies show how the components influence the performance of TraffiCFUS. In the future, our studies aim to design dynamic text-refine schemes for different textual applications and explore a wider range of application possibilities in different realistic scenarios.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 92267104, and Jiangsu Provincial Major Project on Basic Research of Cutting-edge and Leading Technologies, under grant no. BK20232032.