From Diversity to Uniformity: Cross-modal Time Series Modeling with Dependent Channel Grouping

1 Introduction

The proliferation of edge devices generates massive volumes of time series, enabling various time series analytical tasks [Liu et al., 2025b; Liu et al., 2024c], e.g., forecasting [Miao et al., 2024], classification [Tran et al., 2025], and anomaly detection [Xu et al., 2024]. We are seeing remarkable advances in deep learning that are successful at extracting information and creating value from time series [Liu et al., 2025b]. However, most existing methods assume the time series data are homogeneous with a consistent number of variables, which limits their adaptability to deal with the time series exhibiting high heterogeneity with varying feature dimensionalities. Such heterogeneity creates a critical bottleneck for a universal time-series modeling approach to generalize across analytical tasks.

∗Corresponding Author

Recent efforts in time series foundation models have attempted to address the heterogeneity issue [Gao et al., 2024; Cao et al., 2024; Chen and Zhao, 2024], which often treat each variable independently [Nie et al., 2023; Liu et al., 2024c], thereby ignoring the inter-variable dependencies [Gao et al., 2024]. More recently, image-based technologies have emerged to convert time series into visual representations, modeling temporal sequences as two-dimensional structures amenable to computer vision techniques [Wu et al., 2023; Zhong et al., 2025]. Especially, VisionTS [Chen et al., 2025] demonstrates that appropriately converting time series into visual representations can reveal essential time series features, e.g., trend, seasonality, and stationarity, thereby facilitating the capture of temporal dependencies. Nonetheless, image-based methods often fail to preserve the sequential nature and inter-variable relations key to understanding time series [Ni et al., 2024].

In this paper, we consider explicitly combining temporal sequence modeling with spatial image-based representations via multi-modal fusion. The explicit multi-modal integration is expected to digest more comprehensive temporalspatial information for more effective time series modeling. However, it is non-trivial to develop such a method due to the following challenges. First, it is challenging to model inter-variable dependencies among heterogeneous time series. It is expected to preserve intrinsic time series characteristics and enable effective knowledge transfer by aligning heterogeneous time series with varying dimensionalities across tasks. Second, it is difficult to achieve high-quality images with comprehensive information from the original time series. Third, it is important to carefully fuse complementary cross-modal temporal-spatial representations, as improper fusion can lead to excessive reliance on one modality and suboptimal performance. This study addresses the above challenges by developing a cross-modal modeling framework, entitled TimeIG, featuring dependent channel grouping and dual-branch temporal-spatial encoding.

TimeIG achieves a unified modality-aware framework for task-unspecific time series modeling that synergistically combines temporal sequence encoding with spatial-image-based representations through sophisticated cross-modal alignment. Specifically, to accommodate heterogeneous time series with varying numbers of variables, we propose a Dependent Channel Grouping module that identifies representative anchor channels to project diverse inputs into a standardized Kdimensional latent space regardless of original dimensionality. The Dependent Channel Grouping module aims to preserve the intrinsic dependencies among variables while providing a uniform representation for subsequent encoding.

Then, we carefully transform the unified time series into temporal images via tailored temporal imaging, which decomposes time series modeling into three complementary components: periodicity, relational matrix, and phaseamplitude. This decomposition reflects the fundamental mathematical properties of time series data-periodicity captures the cyclical patterns inherent in temporal phenomena, relational matrices encode the correlation structures among variables, and phase-amplitude analysis preserves the frequency domain characteristics crucial for understanding temporal dynamics. Next, we design Dual-branch Encoding, including time series encoding and temporal image encoding branches, to capture temporal dynamics and image-based spatial patterns simultaneously. The time series encoding branch captures sequential dynamics through patch-based Transformer [Liu et al., 2024d], while the temporal image encoding branch learns the effective spatial patterns with imagelike convolutions. he cross-modal fusion challenge is addressed through the proposed Cross-Modal Attention Mechanism (CMAM) combined with Dynamic Weighted-Averaging Mechanism (DWAM), which adaptively determines the optimal integration strategy based on the specific characteristics of each input sequence. Rather than static weight fusion or simple concatenation, these mechanisms enable dynamic interaction between temporal and spatial representations, ensuring that the final fused representation optimally balances the contributions from each modality based on their relevance to the specific task and input characteristics while reducing redundancy and conflicting signals.

The major contributions are summarized as follows:

• We introduce a unified cross-modal time series modeling approach, TimeIG, that addresses the time series dimensional heterogeneity via Dependent Channel Grouping through independent similarity search and adaptive dimensionality alignment mechanisms.
• We propose a novel dual-branch encoding strategy combined with sophisticated cross-modal attention mechanisms that optimally integrate temporal sequence dynamics with spatial structural representations, enabling superior modeling of complex time series patterns.
• Extensive experiments conducted on real-world datasets prove the effectiveness of the proposed TimeIG for universal time series analytics, including prediction, classification and anomaly detection, showing comprehensive performance improvement over the SOTA baselines.

2 RelatedWork

Task-specific Time Series Modeling. Traditional time series modeling methods are mostly task-specific, which are developed for specific time series tasks, such as forecasting [Xu et al., 2025; Liu et al., 2025a], classification [Tran et al., 2025; Liang and Wang, 2024], and anomaly detection [Liu et al., 2024b; Chen et al., 2023]. In the early stage, statistics-based time series modeling methods became mainstream, such as ARIMA [Shekhar and Williams, 2007]. Numerous neural architectures have been developed for effective task-specific time series modeling, including Temporal Convolutional Networks [Cheng et al., 2024], Recurrent Neural Networks [Liu et al., 2021], Multilayer Perceptrons [Ekambaram et al., 2023], and Transformers [Liu et al., 2024d; Liu et al., 2025a; Chen et al., 2023]. However, these methods are mainly invented for specific tasks, falling short in handling different tasks simultaneously.

Universal Time Series Modeling. Universal time series modeling methods aim to develop a universal modeling paradigm that handles different tasks [Liu et al., 2024c; Yin et al., 2025], overcoming the limitations of traditional task-specific models by means of large-scale pretraining and adaptive fine-tuning [Nguyen et al., 2024]. These models often adopt a unified architecture that supports a range of time series related tasks, including forecasting [Nie et al., 2023], anomaly detection [Gao et al., 2024], and classification [Nguyen et al., 2024]. The core idea is to project time series from different tasks into a general feature space to understand their common temporal semantics. Recent studies have sought to develop novel architectures to model diverse time series [Wu et al., 2023; Gao et al., 2024]. However, LLMbased methods require significant computational resources, resulting in high training costs. To be specific, these methods process each channel of the time series individually. However, the correlations across channels are ignored, which may result in significant model performance deterioration.

3 Methodology

Figure 1 shows the framework overview of the proposed TimeIG. As shown in the figure, TimeIG consists of four major modules: Input Projection, Dependent Channel Grouping, Dual-Branch Encoding, and Cross-Modal Alignment. Specifically, Dependent Channel Grouping selects K anchor channels to align varying dimensions into a standardized latent space. Next, Dual-Branch Encoding captures multifaceted features through two parallel branches. Time Series Encoding branch directly encodes the temporal numerical information, while the tailored Temporal Imaging Encoding branch extracts structural correlations by transforming sequences into visual data. Finally, Cross-Modal Alignment adaptively fuses these representations using Cross-Modal Attention (CMAM) and Dynamic Weighted-Averaging (DWAM) to form a unified representation. Next, we provide the technical details of each module, respectively.

integrated via a distance-weighted fusion mechanism. Specifically, first, feature channels are randomly selected, and then the newly added features are connected to the existing features, performing a hierarchical projection. Each channel's time series is mapped onto the closest representative channels weighted inversely by their distance:

where xj is the time series of the channel j, d(·, ·) is a distance metric between time series, and x′j is the fused representation obtained by weighted aggregation. This weighted fusion ensures that the discarded features' information is not completely lost, but softly integrated into the lowdimensional feature space via their nearest representative channels. Through the above procedure, a fixed dimension K representation of the multivariate time series is obtained, preserving the majority of the original variables' information and structural characteristics. Leveraging this structure for nearest neighbor grouping on this reduced space further ensures efficient and effective similarity queries.

3.3 Dual-Branch Encoding

To capture the multifaceted dependencies within the standardized K-dimensional representation X ∈ RB×K×T , we propose a dual-branch encoding module. As shown in the central part of Figure 1 , this module independently models time series encoding branch and temporal image encoding branch, facilitating a comprehensive understanding of complex time series patterns.

Time Series Encoding Branch. Time series encoding branch aims to capture long-range temporal dependencies within a unified, standardized space. Unlike task-specific models that rely on isolated local patterns, our approach employs a Multi-Head Self-Attention (MHSA) mechanism to model the global dynamics of the input patch sequence.

3.1 Input Projection

In this paper, we define the cross-domain datasets as D = {D1,D2, . . . ,DN}, where each domain Di contains multiple time series Di = {T i 1 , T i 2 , . . . , T ini } ∈ RC×L, where C denotes the number of channels and L denotes the length, which together lead to the cross-domain data being heterogeneous. We regard each time series as a channel, and the relationships between different channels are crucial for crossdomain time series models. Most existing studies [Liu et al., 2024c] employ channel-independence mechanisms to accommodate cross-domain time series with different dimensionalities, which ignores the correlations across channels. In contrast, we adopt channel mixing to capture cross-variable interactions and extract more comprehensive temporal patterns. Initially, the original C-dimensional multivariate time series is decomposed into C one-dimensional time series via channel independent processing.

3.2 Dependent Channel Grouping

As shown in the left part of Figure 1, in the dimensionality reduction and fusion process of multivariate time series, we propose a distance-weighted representative feature selection mechanism via Dependent Channel Grouping. To achieve the target of fixed-dimensionality reduction, a subset of K < C most representative time series channels is selected from the channels processed by channel-independent handling as core features. This selection aims to retain the subset with the maximal information content and expressive power among the multivariate data. The selection of the K most influential anchor channels, denoted by the set S, is formulated as an optimization problem:

where R(i) denotes the representativeness score of channel i. For the remaining (N -K) channels, their information is Given the sequence of patches (where M is the total number of patches), the updated temporal representation H is computed as:

where Q,K,V ∈ RM×d are the query, key, and value matrices projected from the complete set of M patches, d is the hidden dimension, dk is the dimension of the key vectors. This global attention mechanism enables the model to perceive dependencies across the entire temporal horizon. To align these sequential features with our group-wise modeling strategy, we introduce a learnable aggregation operator A. This operator distills the high-dimensional temporal information into group-specific features ZTS:

where K denotes the number of dependent channel groups, B is the batch size. This formulation ensures that the learned temporal evolution is both robust and consistent with the cross-modal grouping logic, facilitating a task-aware representation that transcends individual channel variations.

Temporal Image Encoding Branch. To complement the temporal encoding, the temporal imaging encoding branch characterizes the structural interactions among variables by mapping the sequence into a visual modality I ∈ RB×L×K×C . We define a multi-view decomposition operator Ψ that extracts features through three parallel streams reflecting fundamental mathematical properties of the data:

• Periodicity: To capture the multi-scale rhythmicity inherent in complex time series, the operator Hper employs temporal convolutions to discern recurrent motifs and cyclical oscillations. This process yields Zper = Hper(X) ∈ RB×L×K×2 , which encodes the local and global temporal regularities.
• Relational Matrix: Addressing the latent topological dependencies across variables, the spatial operator Hrel computes a dynamic correlation graph. This stream facilitates the modeling of inter-variable synergies, represented as Zrel = Hrel(X) ∈ RB×L×K×1 , providing a structural blueprint of the system's internal interactions.
• Phase-Amplitude: Recognizing that temporal dynamics are often governed by spectral properties, the frequency-domain operator Hfreq decomposes the signal into learned spectral bases. By extracting the phase and amplitude characteristics, it generates Zfreq = Hfreq(X) ∈ RB×L×K×2 , capturing the energy distribution of the signal across the frequency spectrum.

The resulting multi-view descriptors are integrated into a cohesive structural representation ZTI through a synergistic fusion sequence. By employing a channel-wise concatenation followed by a learnable convolutional projection, the model distills the heterogeneous perspectives into a compact structural fingerprint:

Where d′ is the projected feature dimension By explicitly decoupling temporal evolution from cross-variable structural correlations, this branch effectively mitigates the representational bottlenecks inherent in purely sequential architectures and enhances the model's inductive bias toward complex data patterns.

3.4 Cross-Modal Alignment

Following the extraction of decoupled features from the two branches, the framework must synergistically integrate these representations to form a comprehensive embedding. As shown in the right part of Figure 1, we propose an adaptive alignment module consisting of a Cross-Modal Attention Mechanism (CMAM) and a Dynamic Weighted-Averaging Mechanism (DWAM) to achieve optimal feature synthesis based on input-specific characteristics.

Cross-Modal Attention Mechanism. To emphasize complementary information while suppressing modality-specific redundancy, CMAM leverages a dual-stream attention operator to model the latent interactions between temporal evolution ZTS and structural correlations ZTI. We first project these features into a shared latent space via linear transformations ϕTS and ϕTI. The cross-modal interaction is formulated through the Query (Q), Key (K), and Value (V) paradigm:

where the query of one modality retrieves relevant structural or temporal contexts from the other. This bi-directional flow ensures that the fused representation preserves the intrinsic temporal-spatial duality of the data.

Dynamic Weighted-Averaging Mechanism. Since the contribution of each modality varies across different samples and tasks, we introduce DWAM to adaptively determine the fusion intensity. We define a learnable gating function Gθ parameterized by a lightweight multilayer perceptron (MLP). The dynamic weight α is estimated by jointly considering the concatenated global features:

where ⊕ denotes the concatenation operator and σ represents the sigmoid activation. The final fused embedding Ffusion is synthesized as a convex combination of the attentive features:

This mechanism facilitates a sample-wise weighting strategy, enabling the framework to flexibly prioritize either sequential dynamics or structural relationships depending on the input's complexity.

Multivariate Decoder. The fused representation Ffusion is subsequently fed into a multivariate Transformer decoder specialized for task-specific inference. By modeling the longrange dependencies embedded in the synthesized features, the decoder generates the final output Y for various analytical tasks:

where Gϕ represents the task-specific projection head optimized to minimize independent loss objectives L(t) .

3.5 Pre-training and Fine-tuning

Pre-training. The goal of the pre-trained model MΘ is to learn general cross-domain feature representations. For this purpose, we design the objective function as follows:

where z(·;Θ) represents the feature vector representation of a time series, zavg is the average of all domain features, and λ is a weighting coefficient that balances the generation probability and feature consistency. By independently optimizing θt for each task, the framework maintains specialization and high performance tailored to each task's data characteristics and objectives. Meanwhile, the shared multi-modal architecture fosters parameter and representation reuse, enabling effective transfer and robustness in cross-domain, multi-task time series applications. Existing time series foundation models[Liu et al., 2024g; Gao et al., 2024] show that pretraining enhances the model performance since more training data is involved, introducing more useful knowledge. Motivated by this, we pre-train TimeIG with the UTSD4G dataset[Liu et al., 2024g] and then finetune it for new datasets.

Fine-tuning. This work proposes a unified cross-modal framework designed to support multiple heterogeneous time series tasks, including prediction, classification, and anomaly detection. Although the framework enables a shared representation and fusion mechanism across tasks and domains, each task's model parameters θt are optimized independently to accommodate their distinct objectives and data distributions.

Formally, for each task t, given its corresponding dataset Dt = {(X(t) i , y(t) i )}Nt i=1 , the training objective is to minimize the task-specific loss:

where fθt (·) denotes the model inference for task t. In crossdomain time-series analysis, a cross-domain dataset is defined as D = {D1,D2, . . . ,DN}, where each domain Di contains multiple time series Di = {T i 1 , T i 2 , . . . , T ini }.

4 Experiments

In this section, we systematically evaluate the efficacy of TimeIG across various tasks within the time series domain, complemented by ablation experiments that elucidate the individual contributions of its components to the overall performance (refer to Section 4.5). To rigorously assess the generalizability of the TimeIG approach, we conduct extensive empirical analyses across three critical time series tasks: Time Series Forecasting (see Table 1), Anomaly Detection (see Table 2), and Classification (see Table 3).

4.1 Datasets and Experiment Setup

Datasets. The experiments are carried out on certain realworld time series datasets. In terms of forecasting, we employ 8 datasets, including ETTh1, ETTh2, ETTm1, ETTm2, ECL, Trafffc, Weather [Wu et al., 2021], and Solar [Liu et al., 2024d]. For anomaly detection, we adopt 5 datasets,including SMD, PSM, SWaT, MSL, and SMAP [Liu et al., 2024a]. For time series classification, we employ UEA [Bagnall et al., 2018].

Baselines. We compare TimeIG with the following existing baselines, including 13 time series forecasting baselines, i.e., ARIMA [Stellwagen and Tashman, 2013], DLinear [Zeng et al., 2021] TimesNet [Wu et al., 2023], PatchTST [Nie et al., 2023], N-HiTS [Challu et al., 2023], iTransformer [Liu et al., 2024e], TimeVLM [Zhong et al., 2025], TimeLLM [Jin et al., 2024], AutoTimes [Liu et al., 2024f], UniTime [Liu et al., 2024c], UniTS [Gao et al., 2024], Timer [Liu et al., 2024g], and TimerXL [Liu et al., 2025c]. For anomaly detection, we compare TimeIG with ARIMA [Stellwagen and Tashman,

2013], FEDformer [Zhou et al., 2022], Informer [Zhou et al., 2021], DCdetector [Yang et al., 2023], Autoformer [Wu et al., 2021], DLinear [Zeng et al., 2021], TimesNet [Wu et al., 2023], Series2graph (Ser2graph) [Paul Boniol, 2020], TranAD [Shreshth Tuli, 2022], IMDIFFUSION [Chen et al., 2023] and Timer [Liu et al., 2024g]. For time series classification, 13 baselines are selected, i.e., LSTM [Shi et al., 2015], LSTNET [Lai et al., 2018], Informer [Zhou et al., 2021], FEDformer [Zhou et al., 2022], Full Attention (Attn) [Haoqing Wang, 2023], [Dempster et al., 2020], InceptionTime [Ismail Fawaz et al., 2020], TCN [Bai et al., 2018], LIGHTTS [Campos et al., 2023], DLinear [Zeng et al., 2021], TimesNet [Wu et al., 2023], UniTS [Gao et al., 2024], and Timer [Liu et al., 2024g]. Please note that TimeLLM is a large language mode based time series analytics method, while Timer, TimerXL, and UniTS are time series foundation models. We follow the default hyperparameter setting in the original paper or the associated code of baselines, enabling fair comparison.

Evaluation Metrics. We adopt mean squared error (MSE) and mean absolute error (MAE) as evaluation metrics for time series forecasting [Qiu et al., 2024]. The F1-score (F1), AUCROC, and PATE [Ghorbani et al., 2024] are adopted as main evaluation metrics for time series anomaly detection. Additionally, F1 and accuracy are used to evaluate time series classification.

4.2 Time Series Forecasting

Time series forecasting is a central task. We evaluate TimeIG against 13 baselines on standard benchmarks (ETTh, ECL, Weather, Traffic, Solar). Time series foundation models (TimeIG, Timer, TimeXL, UniTS) are trained and tested simultaneously on all datasets. As shown in Table 1, TimeIG achieves the best performance in most cases across prediction lengths from 96 to 720, outperforming the best baseline by up to 15.9%. ARIMA performs worst, as traditional statistics-based methods are shallow and fail to capture complex temporal correlations. Moreover, time series foundation models generally outperform both transformer-based (e.g., PatchTST) and MLP-based (e.g., TimesNet) methods, demonstrating their superior generalization capabilities.

4.3 Anomaly Detection

Since the anomalies are usually hidden in the large-scale data, making the data labeling hard, we focus on unsupervised time series anomaly detection, which is to detect the abnormal time points. We evaluate unsupervised point-wise anomaly detection on five benchmarks (SMD, MSL, SMAP, SWaT, PSM) spanning service monitoring, space telemetry, and industrial control. Following Anomaly Transformer, we use fixed-length sliding windows and train via reconstruction. We report F1, AUC, and PATE as our primary metrics as prior works [Ghorbani et al., 2024; Liu et al., 2024a].

As shown in Table 2, TimeIG attains the highest average performance across five benchmarks on F1/AUC/PATE (80.20/85.31/75.73). It ranks first on SMD and PSM across all three metrics; on SMD, TimeIG reaches 12.4% absolute gain in terms of PATE over the second-best IMDIFFICTION. These results demonstrate that TimeIG provides stable crossdomain generalization and effective range handling in unsupervised anomaly detection.

4.4 Classification

We conduct time series classification experiments on 10 UEA subsets using Accuracy and F1-score. The overall average results are in Table 3. TimeIG outperforms all baselines and achieves the best results on all 10 datasets, demonstrating its effectiveness. Among baselines, the foundation model Timer achieves the best performance, surpassing the previous SOTA by 1.4% and 0.6%, highlighting the potential of foundation models. TimeIG improves over Timer due to its dual-branch encoding module, which learns effective representations across two complementary modalities. This dualbranch approach enhances feature extraction and provides a more robust classification framework by integrating diverse information sources. The comparative analysis in Table 3 further confirms TimeIG's superior accuracy and F1-score, underscoring its adaptability and reliability on complex time series data.

w/o_CMA TimeIG

(a) ECL

w/o_CMA TimeIG

(b) Traffic

4.5 Ablation Study

To understand the effects of TimeIG's different components, we evaluate three components: 1) w/o TS: TimeIG without time series encoding branch; 2) w/o Image: TimeIG without time image encoding branch; 3) w/o CMA: TimeIG without cross-modal attention mechanism. Figure 2 shows results for forecasting (a,b) and anomaly detection (c,d). Regardless of the datasets, TimeIG outperforms its counterparts, showing that these three components are all useful for effective universal time series modeling. TimeIG obtains MSE and MAE reductions up to 38.6% and 22.5%. Further, on all datasets, w/o TS performs worst among all variants. TimeIG performs better than w/o TS by at least 8%, which indicates the importance of time series encoding branch.

4.6 Case Study

To intuitively show the effectiveness of the proposed TimeIG, we provide case studies on Traffic and SMD in terms of forecasting and anomaly detection, respectively, as shown in Figure 3. In Figure 3(a), we see that the predictions are highly consistent with the ground truth, demonstrating the effectiveness of TimeIG. Figure 3(b) shows that TimeIG successfully identifies the outliers on SMD, demonstrating its superior performance for anomaly detection. These two figures jointly demonstrate that the TimeIG model can accurately predict future trends and promptly detect anomalies when handling different time series analysis tasks.

4.7 The Effect of Hyper-parameter K

We next investigate the effect of the number of K on model performance, which denotes the number of selected representative channels. We vary K within the range of {10, 15, 20, 25, 30, 35} to evaluate its sensitivity. As shown in Figure 4, we observe that the F1 and AUC curves first increase, then drop, and finally increase slightly. TimeIG achieves the best performance when K is set to 20, which shows that 20 is the ideal setting in this study. While a larger K increases data coverage, excessive channels introduce noise and redundancy, hindering cross-modal alignment and degrading task generalization.

5 Conclusion

This work presents TimeIG, a modality-aware dual-branch framework for universal time series analysis. To handle heterogeneous dimensionalities, we propose a channel dependency search to select representative channels. For comprehensive feature extraction, we introduce a time series imaging method and a dual-branch architecture for joint sequence and vision representation learning, along with a cross-modal alignment module to fuse complementary features. Extensive experiments show TimeIG achieves state-of-the-art accuracy.

Acknowledgments

This work is partially supported by the National Natural Science Foundation of China (No.62572489) and SCRI, The Hong Kong Polytechnic University (No. Q-CDDG).

Contribution Statement

Minjun Cao and Hao Miao are co-first authors with equal contributions. Senzhang Wang is the corresponding author.