Latent Processes Identification From Multi-View Time Series

1 Introduction

Detecting causal relationships from time series based on observations is a challenging problem in many fields of science and engineering [Spirtes et al., 2000]. A thorough grasp of causal relationships, interaction pathways, and time lags is valuable for interpreting and modeling temporal processes [Pearl, 2000]. Existing works [Chickering, 2002; Tsamardinos et al., 2006; Zhang, 2008; Hoyer et al., 2008; Zheng et al., 2018] typically rely on predefined variables. However, such a strategy is not directly applicable to realworld scenarios, where data are intertwined with unknown generation processes, and causal variables are not readily available. Therefore, identifying the latent sources is crucial for interpreting underlying causal relations and elucidating the genuine dynamics inherent in the temporal data.

∗Corresponding author.

To cope with this problem, non-linear independent component analysis (nICA) [Hyv¨arinen and Pajunen, 1999] has shown promising results in terms of identifiability, by effectively exploiting the underlying structure of the data. On time series data, most nICA techniques attempt to utilize the temporal structure among sequential observations, such as temporal contrastive learning [Hyv¨arinen and Morioka, 2016], permutation contrastive learning [Hyv¨arinen and Morioka, 2017], and generalized contrastive learning [Hyv¨arinen et al., 2019]. However, these methods inherently assume the latent components are independent, which may not hold in practice. Moreover, in broader real-world scenarios, causal discovery tasks mostly require identifying dependent relations from extensive data collections, which may further restrict the identifiability of nICA techniques. Take Figure 1 as an example, understanding the physiological processes of human actions requires jointly analyzing the data across multiple kinds of sensors and a physiological signal may influence multiple downstream regions with different time lags. Similar cases also exist in a wide variety of real-world scenarios, such as multi-view sensors information fusion [He et al., 2021], multi-market stock index analysis [Wang et al., 2020], etc. Hitherto, few efforts have been made to address such challenging yet realistic problems.

The example above, which we call the multi-view latent process identification (MVLPI) problem, poses two combinatorial challenges. First, the diverse time delays in interactions among variables in time series data complicate direct estimation of latent variables. Second, different views typically correlate with subsets of latent factors. Due to identifiability issues, merely inverting these views does not guarantee alignment of the recovered factors. Hence, aggregating them to the final complete latent variables is non-trivial.

In this paper, we propose the Multi-view LatenT Processes Identification (dubbed MuLTI) framework that learns identifiable causal related variables from the multi-view data. MuLTI identifies latent variables from multi-view observations using three strategies: 1) It reformulates causally related process distributions using conditional dependence, replacing original latent variables with independent causal process noises; 2) It applies contrastive learning to maximize mutual information between prior and posterior conditional distributions; 3) It aligns partially overlapping latent variables using learnable permutation matrices, optimized with the Sinkhorn method. The first two strategies ensure the learning of identifiable latent variables, while the last one bridges the causal relations across multiple views. Given that only a subset of each view's source components correspond, matching and merging shared latent variables resemble the sorting and alignment of top-k shared components among estimated variables.

We evaluate our method on both synthetic and real-world data, spanning multivariable time series and visual tasks. Experiment results show that the latent variables are reliably identified from observational multi-view data. To the best of our knowledge, the learning of causal dependent latent variables from multi-view data has no prior solution. The proposed framework may also serve as a factor analyzing tool for extracting reliable features for downstream tasks of multiview learning. More theoretical and empirical results can be found in Appendix 1 .

2 Methodology

2.1 ProblemFormulation

In contrast to the single-view setup, the MVLPI problem presents a unique challenge: only a portion of the latent factors can be recovered from an individual view. Thus, it is necessary to revisit the data-generating procedure of the MVLPI task. As shown in Figure 2, we consider the MVLPI task on the time series data, with the goal of recovering the latent factor zt ∈ Rd at each time step t, which uniquely generates the observed views. To achieve this, we make a common assumption that the current state is spatiotemporally dependent on the historical states,

where Pa(zi,t) denotes causal parents of i-th factor at current step t, ϵi,t denotes a noise component of causal transition. While zt contains a complete set of latent factors that we are truly interested in, we note that each observed view may be dependent on only a subset of it. Formally, we have the following data-generating process,

x1t = g1 (z1 t ), x2t = g2 (z2 t ), where zt = z1 t ∪ z2 t , (2) where xvt is the v-th observed view at time step t, and gv (·) is the corresponding map function. Note that here we consider a 2-view setup for the sake of brevity, while our framework can be easily generalized to handle more views.

1 https://arxiv.org/abs/2305.08164

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

8

q

r

j

0

V

3

k

K

I

p

Q

W

X

E

O

J

H

Y

>

A

B

/

c

D

L

S

g

N

z

G

M

y

C

2

T

P

Z

w

n

d

v

o

7

R

u

+

m

F

9

5

f

U

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

z

8

j

W

v

E

T

S

5

+

U

A

C

q

n

7

O

Y

>

D

H

c

V

L

N

F

J

3

2

d

u

g

k

X

o

p

I

Z

y

r

0

m

P

M

R

9

Q

K

/

B

G

w

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

m

u

5

X

J

K

O

r

y

g

I

p

o

3

j

P

k

V

Y

+

w

>

A

C

c

7

S

N

B

F

G

M

z

Q

0

H

Z

2

d

f

T

9

E

/

U

8

n

R

q

W

D

v

L

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

S

F

E

I

Z

g

A

T

C

N

3

R

m

7

P

K

>

D

H

c

V

L

J

U

2

v

d

u

B

o

Q

j

y

r

0

k

z

M

8

+

O

/

p

X

9

n

q

W

f

w

G

Y

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

c

m

r

z

k

o

8

d

Y

M

G

3

R

F

A

>

C

V

7

S

g

N

B

L

0

X

+

p

2

I

9

U

u

j

W

E

O

T

w

y

v

Z

/

H

Q

f

J

D

P

K

n

5

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

m

W

G

r

2

U

A

J

0

Y

3

k

z

L

o

T

v

>

C

n

c

V

D

S

N

F

d

u

B

Q

E

X

j

K

9

g

P

p

I

H

/

7

q

w

O

f

y

8

Z

R

M

+

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

2

Y

I

H

C

/

A

R

M

0

g

P

r

E

w

n

3

>

X

c

V

9

S

N

B

J

L

z

F

+

o

O

Q

K

y

q

G

j

D

k

u

v

W

T

7

f

d

m

8

Z

p

U

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

R

V

w

8

0

3

N

K

S

2

P

k

D

M

I

>

A

C

B

X

c

7

g

F

L

r

Z

J

m

p

q

W

O

Y

y

u

T

z

v

E

/

9

n

o

H

d

Q

G

U

+

j

5

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

z

p

Y

7

/

q

+

f

3

d

B

2

S

m

J

5

n

g

X

c

>

A

C

V

N

F

L

0

G

o

I

D

j

K

U

E

k

y

8

v

M

r

W

P

R

Q

w

u

H

T

Z

9

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

p

f

3

W

7

S

G

9

U

A

N

E

k

0

8

R

j

z

T

o

>

C

B

X

c

V

g

F

L

r

M

Z

J

m

w

q

I

O

Y

y

D

u

2

v

/

P

n

K

H

d

Q

+

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

5

u

w

J

9

0

j

8

W

S

F

V

M

f

z

B

3

T

Q

>

A

C

X

c

7

g

N

L

G

+

o

p

Y

2

I

D

q

K

U

E

m

k

y

v

r

P

n

R

H

/

Z

d

O

Following the above definition, we can identify the factors from the observed views, and aggregate them into the complete version zt. However, as each view performs a distinctive nonlinear mixing of the source factors, it is challenging to identify the corresponding latent variables entangled in views or to ensure the factors from different views are in the same subspace. Second, though we slightly abuse the notation '∪' to show the aggregation relation of view-specific and the complete factors, it is non-trivial to reconstruct the complete zt from a set of indeterminate estimations in practice. To remedy this problem, we introduce our novel MuLTI framework which comprises a contrastive learning module for enhanced identifiability alongside a novel merge operator that aggregates the view-specific latent variables to a complete one. The whole procedure is illustrated in Figure 3.

2.2 Contrastive LearningModule

Our goal is to learn an inverse function r(·) from the observed data. This is achieved by: (1) learning individual inverse functions rv : X v ↦→ Z v ; (2) introducing a merge function m : {Z v } ↦→ Z, such that r(·) = m({rv (xv )}). Note that each rv (·) corresponds to an inverse of gv (·). We will introduce the merge operator in Sec 2.4, here we omit the superscript v to represent the merged functions and source variables. Formally, the ultimate objective is to optimize r(·) to create a consistent mapping, h = r ◦g, between the estimated ˆ zt = h(zt) and the true zt, e.g., an isometry transformation. We exploit the spatiotemporal dependencies of zt to infer the values of latent variables from preceding states, as shown in Eq. (1). To achieve this, we introduce the causal transition function f(·) to approximate the inherient conditional distribution p(zt|zHt ), where zHt denotes the set of preceding states of zt. Empirically, we estimate an alternative value of the complete latent factor by ˜ zt = f(zHt ).

Motivated by the results of nICA [Hyv¨arinen et al., 2019; Zimmermann et al., 2021], we can identify the underlying factors of data with a certain low indeterminacy by carefully exploiting the inherent structure of data via a variety of conditional distributions. To this end, we can achieve model identifiability - that is, the ability to exactly recover the variables zt and their associated causal relations - by learning the function r, in such a way as to ensure h is an isometry. Specifically, h : Z ↦→ Z must statisfy the condition δ(zt, ˜ zt) = δ(h(zt), h(˜ zt)) everywhere, where δ(·, ·) is a metric. Using the conditional dependency in sequential latent states {zt}, a key insight is to estimate the function f for causal relations modeling and estimate reverse function r:

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

V

j

P

d

n

w

f

E

S

+

5

3

k

u

7

2

9

K

R

U

>

A

C

c

D

L

N

F

J

v

W

B

o

Q

G

X

g

p

0

M

z

/

Y

8

H

r

O

I

q

T

Z

m

y

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

A

C

c

r

9

0

D

G

W

Y

S

y

F

k

5

M

V

g

f

E

>

3

L

N

J

P

U

d

I

u

n

B

q

T

K

m

7

+

j

z

X

R

o

v

/

w

H

8

2

Q

p

O

Z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

I

f

q

Q

N

g

T

u

8

G

C

w

M

z

R

r

n

0

>

A

c

V

D

L

S

F

J

3

P

W

Y

B

H

p

X

j

K

9

k

E

/

7

+

Z

m

v

5

2

U

d

y

O

o

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

U

T

9

V

n

X

B

o

y

J

M

S

/

O

d

I

v

>

A

C

c

D

L

N

F

2

u

Q

R

Z

r

0

w

m

k

8

g

f

+

p

P

E

q

7

G

5

z

Y

j

K

W

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

K

C

5

L

U

r

z

F

m

A

c

P

N

R

3

X

q

W

S

w

p

o

>

B

+

H

V

9

E

J

M

8

v

Y

T

Q

Z

g

y

f

n

d

O

0

k

G

2

D

I

7

/

u

j

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

P

5

U

A

n

v

C

Y

2

E

z

c

3

0

q

L

>

H

V

7

T

M

w

F

I

K

u

W

B

o

j

y

k

X

D

8

/

Z

O

G

S

+

9

Q

d

g

p

N

f

m

R

r

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

E

L

H

d

F

X

w

S

C

Q

j

z

Z

r

v

M

>

A

c

V

D

N

J

3

P

U

O

R

G

0

o

W

9

Y

p

+

m

T

/

7

u

n

g

f

y

2

8

K

B

I

q

5

k

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

f

9

z

B

o

G

w

d

I

8

7

Q

v

+

g

H

3

V

n

>

A

C

X

c

D

L

S

N

F

J

U

2

q

E

0

r

m

M

T

/

W

Z

5

K

R

j

Y

y

u

P

O

p

k

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

8

O

A

2

f

p

R

C

7

D

5

d

L

y

Y

3

g

>

B

H

c

V

S

N

F

J

r

u

w

E

K

q

G

X

j

z

m

Q

0

v

W

/

9

Z

I

P

n

o

k

T

M

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

8

u

E

q

z

p

r

W

v

O

c

U

J

L

M

D

k

F

>

A

C

3

V

S

N

P

9

d

R

X

K

g

H

2

5

I

/

n

T

7

w

f

y

G

o

Y

0

B

j

Q

m

+

where ˜ zt = f(zHt ) is the estimated latent variable based on the previous latent variables zHt , I(·; ·) indicates the mutual information. We denote the prior distribution as p(˜ zt|zHt ) and the modeled distribution as qh,f (˜ zt|zHt ). The optimization problem defined in Eq. (3) can also be viewed as a minimization problem for the cross-entropy H(·∥·),

Nevertheless, directly optimizing the Eq. (4) is typically difficult. To this end, we employ contrastive learning as a surrogate, given that it has been proven to be a variational bound of mutual information maximization [van den Oord et al., 2018]. Formally, the contrastive loss is defined as follows,

M ∈ Z+ represents the fixed number of negative samples, pdata denotes the distribution of all observations, and ppos signifies the distribution of positive pairs, we choose the ℓ1 as the metric, represented by δ(·, ·), µ ≥ 0 represents the temperature. Here, we omit the subscript t for simplicity.

As mentioned earlier, we can infer the latent variable zt in two ways: directly, with ˆ zt = r({xvt }), and indirectly, with ˜ zt = f(zHt ). Given the expectation that these two inferred variables should be similar, we define the pair (ˆ zt, ˜ zt) as a positive pair. Subsequently, we sample data from the marginal distribution of observations, estimate latent variables zt , and form negative pairs with ˆ zt. At this point, the contrastive learning minimizes the cross-entropy between the ground-truth latent conditional distribution p(zt|zHt ) and a specific model distribution qr,h(zt|zHt ). Thus, we have δ(h(zt), h(f(zHt ))) ∝ δ(zt, f(zHt )) for all zt and f(zHt ). This implies that the ground-truth zt lies in the isometric space of the estimated ˆ zt, and can be further recovered through simple transformations.

2.3 Parametric Causal Transition

To represent the causal transition process of f(·), we consider a widely-used parametric formulation that aligns well with the Granger causality [Ding et al., 2006]. To parameterize the dependence of zi,t on its causal parents Pa(zi,t) in Eq. (1), we model the causal transition as following vector autoregressive (VAR) process. Let Aτ ∈ Rd×d be the full rank state transition matrix at lag τ. Assuming the true zt is known, the causal transition can be represented as follow,

here, L is max time lag, ϵt is an additive noise variable. Based on this formulation, we can reformulate the conditional distribution p(zt|zHt ) by changing the variables:

As the ground-truth zt is not readily available, we reuse the estimated ˆ zt to calculate Eq. (6). Through this reformulation, we can represent the causal transition using a mutually independent noise distribution.

Through the above proper definitions of the contrastive pairs and the causal transition f, minimizers of contrastive loss determine the h up to an isometry:

Theorem 1 (Minimizers of contrastive objective recover the latent variables and their relations). Let latent space Z be a convex body in Rd , h = r ◦ g : Z ↦→ Z, f be a causal transition function, and δ be a metric, induced by a norm. If g is differentiable and injective, r, f are expressive enough to be minimizers of contrastive objective Lcontr in Eq. (5) for M → +∞, then, we have h = r ◦ g is invertible and affine mapping and f captures the ground-truth causal relations.

Theorem 1 guarantees the identifiability of the encoder r and causal transition function f (please refer to Appendix C for the proof). Specifically, the contrastive loss enforces δ(h( ˜ zt), h(zt)) ∝ δ(˜ zt, zt) almost everywhere, which leads h to be an isometry, i.e., there exists an orthogonal matrix U ∈ Rd×d such that ˆ zt = Uzt and ˆAτ = AτU ⊤. Further, when ϵt is non-Gaussian (e.g., Laplacian), and metric

δ(·, ·) is non-isotropic (e.g., ℓ1 norm), then there is only a channel permutation π between estimations and ground-truth such that ˆ zi,t = sizπ(i),t, where si is a scaling constant.

2.4 Merge Operation

As mentioned earlier, a significant challenge in learning a spatiotemporal dependent source frommulti-view data lies in aggregating the view-specific source, zv t , into the complete latent source, zt. This is because each view only holds a portion of the total information. Given that sources estimated from each view exhibit unique uncertainties, ensuring their alignment within the same subspace to yield the desired complete source presents a non-trivial task. Yet, given that a portion of the information is shared among different sources, it is feasible to establish transformation between them by identifying a common source. We subsequently cast the problem of identifying a common source as one of the optimal transport objectives, which involves searching plans to move factors from view-specific sources to the common source.

Permutation learning. Assume there is a common source ct ∈ Rdc shared across all view-specific sources zv t . While directly searching for the correspondence of common components can be infeasible, we propose to enforce the cvt be in the top-dc channels of every zv t via permutation learning. Formally, we can achieve this by multiplying the zv t with a doubly stochastic matrix Bv ∈ Bdv as follows,

After that, we can impose the permutations that generate embeddings whose top dc entries are most correlated:

In essence, our goal is to ensure that all extracted instances of {cvt } have entries closely resembling those of the identified common components. Notably, the procedure outlined above equates to regularizing these {cvt } so that they are concentrated around their mean. To achieve this, we first estimate a mean center from the currently extracted common sources:

In practice, we achieve the objective of Eq. (9) by updating the transport plans {Bv } and minimizing transport cost from top-dc view specific components to c∗t in an alternating manner:

where R(Bv ) = - ∑i,j Bv i,j log(Bv i,j ), η > 0 is a constant. This criterion effectively results in learning that c1t = c2t , which represents the ultimate solution for the merging of latent variables.

After this reformulation, with the center c∗t fixed, we can search for the locally optimal {Bv} separately. Crucially, each sub-problem now becomes a standard combinatorial assignment problem [Peyr´e and Cuturi, 2019], which involves the search for optimal transport plans that assign common components from view-specific sources to the common source c∗t . Ultimately, by iteratively applying the center mean estimation procedure from Eq. (10) and utilizing the Sinkhorn algorithm, we can achieve the minimizer of our final objective, as expressed in Eq. (11) (see Appendix B for additional details). We can consider the application of the transformation Bv as a process of smooth component sorting. When we can identify estimated latent sources up to a permutation transformation, we can reduce the doubly stochastic matrices Bv to permutation matrices, i.e., {P v |P v ∈ {0, 1}dv×dv }. This resolution addresses the issue of noncorresponding components.

For merge operator m, we concanate the c∗t with all remaining private components:

Consequently, we can input the merged latent source into the contrastive learning and the inference of causal relations, as outlined in Sec 2.2, thereby completing the entire objective, which is stated in the following theorem (please refer to Appendix C.3 for the proof):

Theorem 2 (Minimizers of the multi-view objective maintains the channel corespondency). Let Z v ⊆ Rdv , Z = ∪Z v and ∩Z v ̸= ∅. If Bv ∈ Bdv be a doubly stochastic matrix, r, f, {Bv } are the minimizers of contrastive objective for M → +∞, then, we have h = r◦g is an isometry, and {Bv } can rearrange the components of each view source such that common components from each view source are aligned.

3 Optimization

We use deep neural networks to implement f(·) and rv (·). The entire optimization process primarily involves two steps: implementing contrastive learning on the estimated sources and learning permutations for the view-specific sources. We jointly train view-specific inverse function networks as well as a causal transition network. Moreover, the permutation matrices {Bv } are optimized alternatively with the main networks during this process.

Enhance noises distribution learning. Contrastive learning is designed to minimize the distance between positive pairs, in other words, it brings ˆ zt and f(ˆ zHt ) closer together. To emphasize this property, we further employ an objective to minimize the residual as:

In parallel, to enhance the mutual independence property of noise ϵi,t, we follow the approach of [Yao et al., 2021] and create a discriminator network D(·) implemented by MLPs. This network is used for discriminating the {ˆ ϵi,t} and randomly permutated versions {ˆ ϵi,t}perm ,

Combining all these elements with weights, the objective is:

For stable training, we perform the optimization of Lcontr and Lm alternately.

4 Experiments

In this section, we present our experimental results on three multi-view scenarios to validate the superiority of MuLTI. More empirical results can be found in Appendix D.

4.1 Datasets

We use synthetic and real-world datasets for evaluating the latent process identification task. Different view-specific encoders are used in different datasets for extracting latent variables. In what follows, we briefly introduce the datasets.

Multi-view VAR is a synthetic dataset modified from [Yao et al., 2021]. To generate the latent process, we first sample transition matrices {Aτ }Lτ=1 from a uniform distribution U[-0.5, 0.5], and then sample initial states {zt}Lt=1 from a normal distribution to generate sequences {zt}Tt=1 . To generate multi-view time series, we randomly select d1 and d2 dimensions of latent variables for each view. The nonlinear observations of each view are created by mixing latent variables with randomly parameterized MLPs gv (·) following the previous work [Hyv¨arinen and Morioka, 2017].

Mass-spring system is a video dataset adopted from [Li et al., 2020] specifically designed for a multi-view setting. A mass-spring system consists of 8 movable balls; the balls are either connected by springs or are not connected at all. As a result of random external forces, the system exhibits different dynamic characteristics depending on its current state and internal constraints. We create two views for this video dataset, as shown in Figure 4, in each of which only 5 of the 8 balls are observable. Consequently, each pair of views includes two videos, each with a duration of 80 frames. This dataset contains 5,000 pairs of video clips in total.

Multi-view UCI Daily and Sports Activities [Altun et al., 2010] is a multivariate time series dataset comprising 9,120 sequences, capturing sensor data for 19 different human actions performed by 8 subjects. Each sample activity contains 125 time steps, each captured by nine sensors, yielding a total of 45 dimensions. We construct the multi-view setting following the [Li et al., 2016]. The 27 dimensions located on the torso, right arm, and left arm are collectively regarded as view 1, while the remaining 18 dimensions on the left and right legs constitute view 2.

4.2 Evaluation Setup

To measure the identifiability of latent causal variables, we compute Mean Correlation Coefficient (MCC) on the validation datasets for revealing the indeterminacy up to permutation transformations, and R2 for revealing the indeterminacy up to linear transformations. We assess the accuracy of our causal relations estimations by comparing them with the actual data structure, quantified via the Structural Hamming Distance (SHD) on the validation datasets.

Baselines. We compare our method with following baselines: (1) BetaVAE [Higgins et al., 2017] neither accounts for the temporal structure nor provides an identifiability guarantee; (2) SlowVAE [Klindt et al., 2021], PCL [Hyv¨arinen and Morioka, 2017], and GCL [Hyv¨arinen et al., 2019] identify independent sources from observations; (3) ShICA [Richard et al., 2021] identifies the shared sources frommulti-view observations; (4) CL-ICA [Zimmermann et al., 2021] recovers the latent variables via contrastive learning; (5) LEAP [Yao et al., 2021] utilizes temporally engaged sources and causal mechanism but only suitable single view data.

For multi-view time series adaptation of baseline methods - β-VAE, SlowVAE, PCL, CL-ICA, and LEAP - we concatenate the series into a single-view format, aligning them by the feature dimension. In the GCL setting, we designate as positive pairs the views originating from the same source, and negative pairs those stemming from distinct sources.

Implementation details. For the Multi-view VAR dataset, we use architectures composed of MLPs and Leaky ReLU units. Ground-truth latent variable dimension d is set to 10, the noise distribution is set to Laplacian(0, 0.05). We set the batch size to 2400, employ the Adam optimizer with a learning rate of 0.001, and utilize β1 = 0.01, β2 = 0.01, β3 = 1e - 5. To verify the influence of the overlapping ratio of views, we select dc ∈ {10, 4, 2, 0} for evaluation. Here, dc = 10 and dc = 0 indicate situations where the latent variables for generating views are fully overlapped and have no overlaps, respectively. To ensure a fair comparison, all baseline methods utilize similar encoders, and the time-lag L of the causal transition module is set to equal the ground truth.

For the Mass-spring system, we create view-specific encoders following [Li et al., 2020], which has similar architectures to the unsupervised keypoints discovery perception module. We set the time lag L = 2 for the causal transition module as the approximated process corresponds to a massspring system, which is a dynamic system of second order. The dimension of estimated latent variables is set to be the same as ground-truth, i.e., 8 coordinates (x, y) account for d = 16. In practice, we first pre-train two pairs of keypoint encoder-decoder in each view. Then, the pre-trained encoders are taken as rv (·) corresponding to each view.

For the Multi-view UCI dataset, we construct two separate encoders with MLPs for two views. The latent dimensions of view 1 and view 2 are set to d1 = 12 and d2 = 9 respectively, the shared latent dimension is set to dc = 3, and the complete latent dimension accounts for d = 18. The max time lag of the causal transition module is set to L = 1. We initially train the MuLTI on the complete dataset, following which we extract the latent variables for downstream tasks.

4.3 Main Results

MuLTI achieves the best identifiability. In Table 1, we report both R2 and MCC for revealing the identifiability of each method. The results highlight MuLTI's superior performance in terms of identifiability across most settings, particularly in the MCC metrics, where it significantly outperforms all other methods. The results indicate that methods like β-VAE, SlowVAE, ShICA, CL-ICA, PCL, and GCL, which primarily depend on the independence of sources and do not model the causal transition, fail to recover the latent variables, even up to linear transformations. LEAP successfully identifies the latent process up to linear transformations using causal transition constraints. Yet, it encounters difficulties when trying to recover the process up to a permutation transformation, especially when observations come from distinct views, despite the conditions being conducive to permutation identifiability.

Table 1: Identifiability results on the VAR process (d = 10, L = 2). Mean+standard deviation over 5 random seeds.

Methods	R2 (%)				MCC
	dc = d	dc = 4	dc = 2	dc = 0	dc = d	dc = 4	dc = 2	dc = 0
β-VAE	40.03±5.31	41.43±4.22	40.67±3.44	38.36±2.71	29.37±8.20	38.31±6.31	31.92±5.61	41.88±4.62
SlowVAE	60.32±5.56	63.21±5.51	62.12±6.13	61.23±3.17	50.32±3.49	51.24±4.71	52.55±2.39	54.21±3.07
PCL	69.78±3.20	80.24±3.32	75.71±2.71	74.56±2.90	52.64±2.33	54.63±2.12	53.42±1.91	55.61±2.62
GCL	73.45±4.34	82.41±3.12	77.65±2.52	73.21±3.31	53.55±1.71	57.32±2.41	52.42±1.70	55.23±2.31
ShICA	41.08±5.76	39.65±4.51	37.61±4.32	38.43±3.12	28.71±5.21	37.21±5.13	29.31±4.97	31.47±3.61
CL-ICA	21.48±6.71	29.37±4.32	24.27±3.33	23.89±2.71	26.79±5.51	33.35±4.51	29.39±5.11	23.68±4.90
LEAP	99.59±0.05	99.67±0.09	99.48±0.07	99.74±0.06	57.56±2.44	62.48±2.12	65.44±1.61	67.75±1.32
MuLTI	99.46±0.09	99.72±0.05	99.68±0.07	99.45±0.08	99.80±0.12	99.27±0.43	99.39±0.09	99.43±0.04

MuLTI successfully identifies the latent process. We show that MuLTI successfully identifies the latent process on both the Multi-view VAR dataset and the Mass-spring system. Figure 5 demonstrates the performance of our MuLTI in estimating the latent process where d = 10, dc = 2. The results suggest that our model nearly perfects the learning of the VAR transition matrices, with the identified latent variables closely resembling the true variables, thus implying a comprehensive identification of the latent process.

The objective of the Mass-spring system task is to estimate latent variables, representing the coordinates of each ball, and transition matrices that elucidate their connections. As displayed in the left panel of Figure 6, the estimated latent variables precisely match the ground-truth, and the recovered transition matrices correspond to the ground-truth ball connections, as indicated by an SHD = 0.

MuLTI improves downstream task. Given that causal variables are often not readily available, directly assessing their identification and relationships in real-world data is challenging. However, under most conditions, identifying the latent process from observations can simplify downstream tasks (e.g., classification and clustering) or directly aid in the extraction of meaningful features. Therefore, we employ a real-world multi-view time series dataset to demonstrate the effectiveness of latent process identification. We employ multivariate time series classifiers - specifically, reservoir computing [Bianchi et al., 2021]. This approach encodes the multivariate time series data into a vectorial representation in an unsupervised manner, allowing us to evaluate learned latent variables {ˆ zt}Tt=1 in the downstream task.

To demonstrate the efficacy of the learned latent representations, we independently train both the baseline methods and MuLTI on the entire dataset, subsequently extracting 18dimensional features to form a new dataset. Then, we train and evaluate the RC classifier on the new datasets extracted by each method. The results are shown in Table 2. We report classification accuracy with varying training set sizes, i.e., we randomly select Ntr ∈ {10, 20, 30, 40, 50} samples per class for training, using the remaining samples for testing. Compared to the baseline methods, the pre-trained features from MuLTI enhance the downstream performance of the RC classifier, particularly for small training set classification tasks.

Table 2: Classification accuracy on Multi-view UCI dataset. Ntr is the number of training samples randomly chosen from each class.

Ntr	raw data	+β-VAE	+LEAP	+ShICA	+MuLTI
10	74.21±0.26	72.53±1.13	73.57±1.90	73.17±0.77	85.59±0.05
20	90.00±0.38	85.82±0.53	86.10±0.86	86.59±0.51	92.14±0.38
30	92.92±0.74	89.37±1.54	88.90±0.69	91.29±0.31	94.18±0.14
40	94.95±0.16	91.16±0.49	91.79±0.07	94.01±0.55	96.15±0.81
50	94.92±0.28	92.53±0.04	93.00±0.03	94.78±0.04	96.82±0.30

Table 3: Ablation results on VAR dataset (d = 10, dc = 4, L = 2).

MuLTI	R2 (%)	MCC
w/o Causal transition	44.67±2.31	33.47±7.60
w/o Permutation learning	99.32±0.07	77.45±0.47
w/o Residuals minimization	99.48±0.04	82.32±2.63
w/o Noise discriminator	99.23±0.05	98.32±0.27

4.4 Ablation Studies

Module ablations. To verify the effectiveness of eachmodule within MuLTI, we conduct ablation studies by individually removing each module. The results tested on the Multiview VAR dataset are shown in Table 3. When the causal transition module is removed, positive pairs are formed from temporally adjacent states, i.e., (ˆ zt, ˆ zt+1). In the setting of removing permutation learning, the view-specific variables are simply concatenated along the feature dimension and mapped with a linear layer to match the dimension of the transition function. Results indicate the crucial role of the causal transition function in identifying the spatial-temporal dependent latent process. As shown, permutation learning also greatly contributes to the identifiability under this multi-view setting. Furthermore, Figure 6 from the Mass-spring system experiment reveals that, without permutation learning, only partial identification of the latent variables is possible.

Effect of noise anddistancemetrics. To evaluate the effect of noise distribution type and distance metric δ(·, ·), we conduct experiments under different combinations. The results shown in Table 4 indicate that the Laplacian noise condition accompanied with ℓ1 norm is key to identifying the groundtruth latent processes up to permutation equivalence. Nevertheless, in most conditions, we can still identify the latent process up to linear equivalence.

5 RelatedWorks

Multi-view representation learning. Many realistic conditions involve data from multiple channels and domains; these data must be analyzed together in order to obtain a reliable estimation of latent relations, e.g. neurophysiology and behavior time series data [Urai et al., 2022]. Typical methods of learning share representations from multi-view data include Canonical Correlation Analysis (CCA) [Hotelling, 1992] and its variants [Akaho, 2006; Andrew et al., 2013; Lai and Fyfe, 2000; Wang et al., 2015; Zhao et al., 2017], but most of these methods only consider learning the common source of each view. More recently, some improvements are done in dividing the sources into common and private components [Lyu et al., 2021; Huang et al., 2018; Hwang et al., 2020; Luo et al., 2022]. However, it is not the case when there are relations (causal influences) between source components, due to most of them relying on independent source assumptions [Gresele et al., 2019]. In contrast, our work seeks to recover latent variables when there are causal relations between them.

Table 4: Identifiability with different noise prior & metric. Note that the Identity indicates the untrained model, and Supervised indicates the model trained with MSE between ground-truth and estimated latent variables. Mean+standard deviation over 5 random seeds.

Noise type	δ(·, ·)	R2 (%)	MCC
Identity	/	66.12±2.77	43.91±2.56
Supervised	/	99.78±0.07	99.92±0.02
Normal (σ = 0.1)	ℓ2	99.72±0.03	67.21±1.41
Laplace (λ = 0.1)	ℓ2	99.71±0.02	66.74±0.21
Normal (σ = 0.1)	ℓ1	99.69±0.04	70.52±0.27
Laplace (λ = 0.1)	ℓ1	99.74±0.02	99.31±0.07
Laplace (λ = 0.05)	ℓ1	99.77±0.03	99.81±0.02

Identifiability. Identifiability means the estimated latent variables are exactly equivalent to the underlying ones or at least up to simple transformation. Traditional independent component analysis (ICA) [Belouchrani et al., 1997] assumes sources are linear mixed, but such linearity relation may not hold in many applications. To overcome this limitation, the non-linear ICA (nICA) [Hyv¨arinen and Pajunen, 1999] has attracted great attention. A variety of practical methods [Hyv¨arinen and Morioka, 2017; Hyv¨arinen et al., 2019; Khemakhem et al., 2020; Sorrenson et al., 2020] have been proposed to achieve identifiability through discriminating data structures, such as manually constructing positive and negative pairs and training models to produce features to distinguish whether input data are distorted, such as temporal structure [Hyv¨arinen and Morioka, 2016] and randomized distorted data pairs [Hyv¨arinen et al., 2019]. More recently, Gresele et al. adopt similar practices into multi-view data, but still limited in source independent assumptions. Yao et al. make attempts on dependent sources but their method can only handle single-view data. In this work, we explore a general case that identifies dependent processes frommulti-view temporal data.

6 Conclusion

In this work, we make the first attempt toward a general setting of the multi-view latent process identification problem and propose a novel MuLTI framework based on contrastive learning. The key idea is to identify the latent variables and dependent structure through the construction of conditional distribution and aggregating variables across views. Empirical results suggest that our methods can successfully identify the spatial-temporal dependent latent process frommultiview time series observations. We hope our work can raise more attention to the importance of identifying and aggregating latent factors in understanding multi-view data.

Acknowledgments

This work is supported by the National Key R&D Program of China (2020YFB1313501), National Natural Science Foundation of China (T2293723, 61972347), Zhejiang Provincial Natural Science Foundation (LR19F020005), the Key R&D program of Zhejiang Province (2021C03003, 2022C01119, 2022C01022), the Fundamental Research Funds for the Central Universities (No. 226-2022-00051). JZ also wants to thank the support by the NSFC Grants (No. 62206247) and the Fundamental Research Funds for the Central Universities.