Conditional Information Bottleneck-Based Multivariate Time Series Forecasting

1 Introduction

Multivariate time series (MTS) forecasting, a focal point in domains such as stock price [Adebiyi et al., 2014], energy management [Li et al., 2024], and weather prediction [Chen et al., 2023], necessitates capturing inter-series correlations that delineate dependencies among variables at the same time and intra-series correlations that signify temporal dependencies within a variable across different time spans. For instance, in weather forecasting, grasping inter-series correlations is critical for deciphering the intertwined relationships among temperature, humidity, and vapor pressure. Higher temperatures induce increased evaporation, resulting in elevated humidity levels and altered vapor pressure. Furthermore, capturing intra-series correlations preserves the temporal coherence of each variable across adjacent time periods, facilitating gradual transitions in temperature between day and night or across seasons.

∗Corresponding author.

Existing methods utilize deep neural networks, including recurrent neural networks (RNN) [Pagliarini et al., 2024], convolutional neural networks (CNN) [Sun et al., 2023], graph convolutional networks (GCN) [He et al., 2024], and multilayer perceptrons (MLP) [Zeng et al., 2023] for MTS forecasting. These approaches often encounter challenges in maintaining temporal coherence due to struggles in capturing long-term dependencies and insensitivity to temporal order. Transformer-based methods [Zhou et al., 2023; Nie et al., 2023] leverage self-attention mechanisms for MTS forecasting. However, they lack interpretability in capturing inter-variable dependencies.

The Information Bottleneck (IB) principle, aimed at retaining relevant information for forecasting while reducing irrelevant information [Tishby et al., 2000], has been applied in IB-based models like LaST and DeepCoupling [Wang et al., 2022b; Yi et al., 2024] to identify temporal patterns while preserving essential forecasting information. However, the exploration of leveraging IB to capture variable dependencies and maintain temporal coherence remains uncharted. Establishing a model that incorporates the IB principle to effectively capture inter / intra-series correlations remains a valuable pursuit.

The conditional information bottleneck (CIB) method integrates task-relevant background information using mutual information through conditional variables, aiming to minimize redundancy while capturing useful information from all data [Choi and Lee, 2024]. Hence, we employ CIB to discern intricate interaction patterns among variables within each time period. Subsequently, to ensure temporal coherence within each variable across neighboring periods, we utilize conditional mutual information (CMI) to preserve the inherent sequence consistency. Addressing the efficient calculation of mutual information and precise representation of correlations in MTS involves tackling the following two pivotal challenges:

• How can the CIB-based principle be formulated to capture inter-series correlations in MTS data?
• How can CMI be utilized to capture intra-series correla-

tions and ensure continuous time coherence?

For the first challenge, we employ CIB to diminish superfluous data while conserving pertinent information for MTS forecasting. To delineate inter-series dependencies, each variable is considered with other variables as conditions, offering additional temporal insights. To streamline mutual information computation within CIB, we establish manageable upper and lower bounds for each mutual information component. Subsequently, we formulate the CIB-based loss function by amalgamating the estimated mutual information terms.

For the second issue, we segment the time series into overlapping patches to enhance the capture of local intra-series correlations within each variable. To preserve the temporal coherence without disrupting the continuity of the time series segmentation, we introduce the CMI-based principle, where all preceding patches are regarded as conditions to offer additional temporal insights. To address the computational intensity of mutual information estimation within CMI, we establish a lower bound for mutual information approximation, subsequently defining the CMI-based loss function.

Ultimately, we introduce a cohesive objective by amalgamating the CIB and CMI-based loss functions to collectively capture inter / intra-series correlations within MTS data. Additionally, we present a training algorithm tailored to MTS data, offering a systematic approach to capturing intervariable dependencies and upholding temporal coherence.

We summarize our contributions as follows:

• We propose the CIB-based principle for MTS forecasting to delineate inter-variable dependencies within concurrent time periods.
• We introduce the CMI-based approach for MTS forecasting to ensure temporal consistency within individual variables across neighboring time segments.
• We present a unified optimization goal and training protocol to concurrently capture both inter / intra-series correlations.
• We conduct extensive experiments on authentic datasets and demonstrate the superior efficacy of our method over state-of-the-art alternatives.

2 RelatedWork

Deep neural network-enabled MTS forecasting. Diverse deep neural network methodologies have emerged for MTS forecasting. RNN-based architectures leverage hidden states to capture temporal dependencies [Li et al., 2022; Pagliarini et al., 2024]. CNN-based models extract local features and recurrent patterns through convolutions [Sun et al., 2023; Wang et al., 2023]. Graph structure is instrumental in capturing intricate dependencies in data [Duan et al., 2024], and thus GCN-based models establish graph structures from MTS, employing graph convolutions to propagate temporal features across nodes [He et al., 2024; Wu et al., 2020]. MLP models are tailored for efficient MTS forecasting [Zeng et al., 2023]. Certain models extract local temporal features by segmenting time series into patches [Huang et al., 2024; Ma et al., 2024]. Nevertheless, these approaches may struggle to capture dependencies over extended periods and might overlook the sequential nature of temporal data, potentially compromising the preservation of long-term patterns crucial for accurate forecasting.

Transformer-based MTS forecasting. To mitigate computational overhead, Zhou et al. [Zhou et al., 2023] introduce Informer, employing ProbSparse attention, while Li et al. [Li et al., 2023] integrate dilated convolutional networks and Transformer blocks to circumvent global attention computations. Liu et al. [Liu et al., 2024] captures inter-variable dependencies by applying self-attention over variate tokens. Strategies such as frequency domain transformation or time series decomposition aid in extracting periodic and global features, easing computational demands [Wu et al., 2021; Zhou et al., 2024]. Techniques like fragmentation [Nie et al., 2023] or multi-scale processing [Zhang and Yan, 2023] of time series, which capture both local and global dependencies. Nonetheless, these approaches lack the interpretability of inter-variable dependencies.

Temporal pattern recognition-based MTS forecasting. Several methods leverage frequency-domain transformations to uncover global and periodic trends in MTS, facilitating the detection of short-term variations and long-term relationships [Yi et al., 2023; Cai et al., 2024]. For instance, Wang et al. [Wang et al., 2022b] utilize mutual information for disentangled representation learning, while Grzegorz et al. [Grzegorz, 2023] decompose MTS into trend and seasonality components to discern distinct patterns. Yi et al. [Yi et al., 2024] identify multi-order dependency patterns and the coupling of MTS through Information Bottleneck (IB) principles. Choi et al. [Choi and Lee, 2024] addresses the issue of overly stringent regularization in IB by introducing conditional regularization based on temporal contexts within time series. Additionally, Liu [Liu, 2022] reduces data dimensions while preserving pertinent patterns in lengthy time series. Ryabko [Ryabko, 2019] maps high-dimensional time series into finite spaces to uncover underlying dependencies accurately. Nevertheless, these approaches struggle with the interpretation of complex multivariate interactions.

3 Preliminaries and ProblemFormalization

In this section, we present formulations of MTS forecasting and conditional information bottleneck.

MTS forecasting. Given L historical time steps of N variables, the input series is denoted as X = {X1,X2, . . . ,XN} ∈ RN×L, where each Xi = {x1i , x2i , . . . , xLi } ∈ RL(1 ≤ i ≤ N) represents the historical values of the i-th variable. In this paper, our objective is to learn a function F : F(X) → ˆX, where ˆX = { ˆX1, ˆX2, . . . , ˆXN} ∈ RN×T denotes the values of all variables in future T time steps, and each ˆXi = {ˆxL+1 i , ˆxL+2 i , . . . , ˆxL+T i } ∈ RT represents the forecasted values of the i-th variable.

Conditional information bottleneck. With random variables X, ˆX and Y representing input feature series, future prediction outcomes, and their corresponding ground truth, the IB principle seeks to ascertain the variable set Z that minimizes irrelevant information from X while upholding crucial information necessary for predicting ˆX that aligns with Y . This is succinctly expressed as:

where I(Z; Y ) and I(Z;X) are the mutual information between Z and both Y and X, respectively. β ∈ R is the Lagrangian multiplier used to balance the trade-off between two terms in IB. Specifically, I(Z; Y ) is formulated as:

By incorporating conditions, the CIB principle is to find the variable set Z that retains the information most relevant to the ground truth Y while reducing irrelevant information from features X under the conditional variables C, formulated as:

where I(Z;X | C) represents the CMI between Z and X given C, formulated as:

The chain rule for CMI is

Accordingly, Eq. (3) could be expanded as:

Problem formalization. Given the MTS X, our objective is to establish the mapping F : F(X) → ˆX, where F represents an encoder designed to generate embeddings that encapsulate the salient temporal details crucial for prediction.

To capture the local intra-correlations in each variable, we divide X into overlapped patches, achieving a segmented representation XP . To reduce redundancy and utilize inter-series correlations when generating embeddings Zi for Xi, we first impose the loss on the encoder to retain useful information for Zi. Then, to capture dependencies among variables, we construct the contrastive learning loss by using positive sample set Z + i and negative sample set Z - i . Thus, the temporal information can be captured from ¯ Xi, which denotes the variables in Xp except Xi. Subsequently, we maximize the CMI between patches within Zi to solve the truncation of the temporal coherence. Additionally, we generate ˆ Xi by Zi and use the Mean Squared Error (MSE) loss between ˆ Xi and its ground truth Yi. Finally, we summarize these steps into an algorithm for MTS forecasting.

4 Methodology

4.1 Framework Overview

We give the framework for MTS forecasting, as shown in Fig.1, including the following components:

• Segmenting time series is proposed to divide time series into overlapped patches, helping the encoder to better capture local intra-series correlations.
• Capturing dependencies among variables is designed and implemented to leverage the CIB principle to identify and capture complex interactions among variables.
• Maintaining time coherence and forecasting is proposed to maintain the intrinsic consistency of MTS by leveraging CMI and then generating forecasting results.

4.2 Conditional Information Bottleneck for Inter-Series Correlation

To capture the dependencies among variables, we employ I(Zi; ¯Xi)(1 ≤ i ≤ N) to quantify the information shared between Zi and ¯Xi, where Zi is generated by qϕ(Zi | Xi) and qϕ(·) denotes the multi-head attention with parameters ϕ.

Maximizing I(Zi; ¯Xi) encourages Zi to capture the dependencies among variables. For this, we adopt the CIB principle centered on I(Zi; ¯Xi), designed to achieve three key objectives: capturing dependencies among variables, minimizing redundancy, and forecasting future observations.

Capturing dependencies among variables. Directly maximizing I(Zi; ¯Xi) for capturing dependencies proves unattainable due to the intricate estimation involved in highdimensional joint distributions. Consequently, we employ the InfoNCE loss [Oord et al., 2018], which establishes a manageable bound to approximate the mutual information:

To construct the InfoNCE loss, we first generate the anchor sample set Z anchor i by selecting Zi within the current batch. Then, we generate the positive sample set Z + i by qϕ(Z + i | ¯Xi), which exclusively contains temporal information from other variables, serving as a reference for capturing dependencies among variables. Positive samples Z + ij = {Z + i1 , Z + i2 , . . . , Z + iN } ∈ Z + i (j ̸= i) consist of all variables in Z + i except the i-th variable, which share the same sample index as Z anchor i .

To prevent Zi excessively focusing on temporal information within its own context Xi, the negative sample set Z - i is generated by selecting Zi from each sample within current batch, where Z - il = {Z - i1 , Z - i2 , . . . , Z - ibs } ∈ Z - i within the current batch represents negative samples, and bs denotes batch size. All the samples are selected exclusively from the current batch, shown as Fig.2. Meanwhile, other labels in Fig.2 refer to the unused data points.

Subsequently, we pair Z anchor i with Z + ij to form positive pairs, and Z anchor i with Z - il to form negative pairs. The similarity between positive and negative pairs can be computed using the Cosine similarity function:

where z1 and z2 are the representations of the feature embeddings of two data samples, respectively.

Finally, we impose the inter-series contrastive loss to approximate I(Zi; ¯Xi) by amplifying the similarity of positive pairs and diminishing the similarity of negative pairs at the batch level.

Redundancy minimization. To reduce the forecastingirrelevant information in Zi, we introduce I(Zi; XP ) to quantify the information shared between Zi and XP . This measure is minimized to prompt Zi to concentrate on pertinent temporal data. Given the intractability of directly minimizing I(Zi;XP ), we employ the Kullback-Leibler (KL) divergence [Kullback, 1997] to establish a feasible upper limit for I (Zi; XP ).

To enable the subsequent use of KL divergence, we first express I(Zi; XP ) using variational probability distributions qϕ(Zi; XP ) and qϕ(Zi) as follows:

Without the distributional constraints, Eq. (11) can still result in overfitting and temporal redundancy in Zi, while qϕ(Zi) is still intractable. Thus, we reformulate I(Zi;XP ) by introducing a standard Gaussian distribution p(Zi) ∼ N(0, I ) into Eq. (11).

To further simplify the minimization of I(Zi; XP ) and impose distributional constraints on Zi, we decompose Eq. (12) into two KL divergence terms:

Given the non-negativity property of the KL divergence, the upper limit of Eq. (13) is:

Eq. (14) can be formulated using the KL divergence and minimized to constrict the information capacity of Zi by promoting qϕ(Zi | Xp) to approximate p(Zi). This process aids in reducing redundancy in the encoding of Zi and minimizing I(Zi;XP ). Therefore, we designate Eq. (14) as the redundancy minimization loss, denoted by Lk.

Future observation forecasting. To encourage Zi to contain more information w.r.t. the ground truth Yi, we formulate I(Zi; Yi) to quantify the shared information between Zi and Yi as follows:

Maximizing I(Zi; Yi) promotes the retention of relevant information in Zi for forecasting Yi. To enable a tractable maximization for the mutual information, we assume that Yi is generated by transforming Zi through a linear transformation model with Gaussian noise ϵ.

where ξ denotes the model's parameters.

Assuming that ϵ ∼ N(0, σ2 ) holds, the conditional distribution of Yi given Zi conforms to a Gaussian distribution (i.e., Yi | Zi ∼ N(fξ(Zi), σ2 ), enabling the direct computation of the conditional entropy H(Yi | Zi) as:

where σ2 is the variance of the Gaussian noise and can be formulated as MSE loss.

By substituting Eq. (18) into Eq. (17) and substituting Eq. (17) into Eq. (15), we have

Since H(Yi) is constant, minimizing MSE(Yi, ˆXi) is equivalent to maximizing I(Zi; Yi). Thus, we give the MSE loss Lm to maximize I(Zi; Yi) in a tractable way.

To establish a unified method for the CIB principle, we give three loss components Lm, Lk, and Lc to minimize I(Zi; Yi), I(Zi;Xp), and I(Zi; ¯Xi), respectively. By integrating these objectives, the unified loss function is formulated as:

4.3 Conditional Mutual Information for Intra-Series Correlation

To maintain temporal coherence in MTS, we give the CMIbased principle, which we aim to maximize for each pair of adjacent embedding patches Zik and Zik-1 (1 ≤ k ≤ M) at the data sample level, where M denotes the number of patches and Zik denotes the k-th patch in Zi. These adjacent patches are conditioned on all preceding patches {Zik-2 , . . . , Zi1 }, formulated as:

In order to optimize Eq. (22), we employ the Mutual Information Neural Estimator (MINE) [Belghazi et al., 2018]. MINE establishes a computationally feasible lower bound for I(X; Z) through the dual form of KL divergence, thereby circumventing the direct computation of high-dimensional joint and marginal distributions.

where {Tθ}θ∈Θ denotes the functions parametrized by a neural network, and ˆP means the empirical distribution associated to n i.i.d. samples.

Constructing a computationally feasible lower bound for I (Zik ;Zik-1 | Zik-2 , . . . ,Zi1 ) directly from Eq. (23) poses a challenge. To tackle this issue, we initially reformulate it in terms of the KL divergence.

Expanding upon this representation, we utilize Eq. (23) to establish a lower bound for the CMI. Through parameterizing the CMI distribution with a neural network-based variational function, the resulting computationally manageable lower bound is articulated as

The first term in Eq. (25) signifies the expectation of Tθ within the joint distribution of {Zik ,Zik-1 ,Zik-2 , . . . ,Zi1 }, reflecting the ability of Tθ to capture the true conditional dependencies present in the data. It assigns greater values when Zik and Zik-1 , given {Zik-2 , . . . ,Zi1 }, exhibit stronger correlation. The second term corresponds to the logarithm of the expected value of eTθ under a factorized distribution approximating a scenario, where Zik and Zik-1 are conditionally independent given {Zik-2 , . . . ,Zi1 }. Specifically, we approximate this factorized distribution by shuffling Zik and Zik-1 while keeping {Zik-2 , . . . , Zi1 } fixed to prevent overestimation of the mutual information between Zik and Zik-1 . By computing the difference between these terms, a tractable lower bound on the CMI is derived, which can be optimized w.r.t. Tθ.

In essence, Eq.(25) furnishes a computationally manageable lower bound for the CMI. This bound is employed for every CMI component within Eq.(22) to delineate the loss function Ls, ensuring the temporal coherence of MTS data.

4.4 Training Algorithm

In terms of inter / intra-series correlations, the larger the interseries correlation, the smaller the inter-series loss. Meanwhile, the larger the intra-series correlation, the larger the intra-series loss. To train the forecasting model, we give the total objective as follows:

where Lm is associated with the forecasting task, encouraging the model to accurately forecast future values. Lk focuses on removing irrelevant information. Lc captures the dependencies among variables, and Ls maintains time coherence of MTS. The hyperparameter α1 serves as a regularization coefficient in CIB, balancing the inter-series temporal interactions and preventing overfitting. α2 regulates the strength of time coherence preservation across patches, encouraging the model to capture consistent temporal patterns while controlling its impact on forecasting performance.

Then, we develop the method simultaneously capturing inter and intra-series correlations, summarized in Algorithm 1. The computation of inter-series correlation from line 4 to line 7 takes O(M · d2 + L · d) time, and the computation of intra-series correlation from line 8 to line 13 takes O(M · d) time. Thus, the total time complexity of Algorithm 1 is O(M · d2 + L · d).

5 Experiments

5.1 Experiment Setup

Datasets. We conduct extensive experiments on 9 realworld datasets, as outlined in [Huang et al., 2024], including ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2), Weather, Traffic, Electricity, ILI, and Exchange Rate. In all experiments, we adopt the same train/val/test split ratio of 6:2:2 for ETT datasets and 7:1:2 for others.

Comparison methods. We compare our method with 8 state-of-the-art methods in 5 categories: (a) Transformerbased models: Crossformer [Zhang and Yan, 2023]. Informer [Zhou et al., 2023]. (b) Linear-based models: HDMixer [Huang et al., 2024]. DLinear [Zeng et al., 2023]. (c) CNNbased model: MICN [Wang et al., 2023]. (d) GNN-based models: MSGNet [Cai et al., 2024]. MTGNN [Wu et al., 2020]. (e) IB-based model: LaST [Wang et al., 2022b].

Metrics. We adopt MSE andmean absolute error (MAE) to evaluate the effectiveness of our method.

5.2 Experimental Results

Effectiveness. We compare the MSE and MAE of all models on various datasets, shown in Table 1, where the input sequence length L is set to 96 for Exchange, 60 for ILI, and 336 for others. The lower the indicators, the better the results. In the table, red bold numbers indicate the best performance, while blue indicates the second best. Based on the results, we observe the following:

• (a) Our method achieves superior performance in 67 metrics, matching the SOTA methods in 3 metrics and ranking second in the remaining 2 metrics.

Algorithm 1 CIB-based MTS forecasting

Input: X: historical MTS data

Parameters: P: length of patches, T: total training epochs, lr: learning rate, ϕ: parameters of encoder, θ: parameters of variational bound, ξ: parameters of forecasting head

Output: ˆX: forecasting results

• 1: Divide X into M overlapped patches Xp
• 2: Initialize ϕ, θ, and ξ
• 3: for t = 1 to T do
• 4: Z ← qϕ(Z|XP )
• 5: Calculate Lk by Eq. (14) // Constraint on encoder
• 6: Generate samples Z + i and Z - i

7: Calculate Lc by Eq. (9) // Inter-series correlations

• 8: Ls ← 0
• 9: for k = 3 to M do
• 10: Calculate Iθ(Zik ;Zik-1 |Zik-2 , . . . ,Zi1 )
• 11: Ls ←Ls + Iθ(Zik ;Zik-1 |Zik-2 , . . . ,Zi1 )
• 12: end for
• 13: Ls ←Ls/(M -2) // Constraint on time coherences
• 14: ˆX ← fξ(Z) // Generating forecasting results
• 15: Calculate Lm by Eq. (20) // MSE loss
• 16: L←Lm +α1Lk +α1Lc -α2Ls
• 17: ϕ←ϕ- lr ∗∇L // Updating parameters
• 18: θ ← θ - lr ∗∇L
• 19: ξ ← ξ - lr ∗∇L
• 20: end for
• 21: return ˆX
• (b) Our method outperforms the existing best results by achieving an overall 4.26% reduction in MSE and a 6.12% reduction in MAE. Moreover, our method achieves an overall 60.2% reduction in MSE and 40.5% reduction in MAE when compared to MTGNN.
• (c) Our method outperforms HDMixer, an advanced patchbased model that improves upon PatchTST, achieving overall reductions of 4.26% / 6.12% in MSE / MAE, respectively.
• (d) Our method outperforms the best existing methods without multivariate correlations by achieving an overall 21.4% and 14.3% reduction in MSE and MAE, respectively.

The consistent improvements across all benchmarks underscore the superiority of our method in delivering accurate results across a wide range of datasets and forecasting settings, as reflected in consistently low error metrics.

Ablation studies. We conduct ablation studies on three datasets by removing specific components from our model. The constraints on the encoder, multivariate information interaction, and smooth data transfer are removed by W/O-Lk, W/O-Lc, and W/O-Ls, respectively. From the ablation studies results in Table 2, we can find that:

• (a) When W/O-Lk is applied, MSE increases by up to 5.1% and averages 4.21%, while MAE increases by up to 4.0% with an average of 2.77%.
• (b) When W/O-Lc is performed, the MSE rises a maximum of 5.6% and averaged 3.43%, while the MAE rises a maximum of 3.2% and averaged 1.87%.
• (c) When W/O-Ls is performed, the MSE rises a maximum of 3.04% and averaged 2.01%, while the MAE rises a

maximum of 2.8% and averaged 1.6%.

These results demonstrate that each component contributes to the model's performance.

6 Conclusion

Our study introduces a method based on CIB for MTS forecasting, aiming to capture both inter / intra-series correlations effectively. CIB is utilized to capture dependencies among variables within manageable bounds, while CMI is leveraged to ensure temporal coherence through a computationally feasible lower bound. Our approach adeptly captures intricate dependencies in MTS data, thereby improving forecasting accuracy compared to traditional methods.

In forthcoming research, we plan to explore the structural relationships inherent in MTS data and design advanced optimization techniques to better capture temporal and crossvariable dependencies for improved forecasting.

Acknowledgments

This paper was supported by the Joint Key Project of National Natural Science Foundation of China (U23A20298), Key Project of Fundamental Research of Yunnan Province (202401AS070138), and Program of Yunnan Key Laboratory of Intelligent Systems and Computing (202405AV340009).