Bridging Causality and Learning: How Do They Benefit from Each Other?

1 Introduction

The main goal of statistics and machine learning is to discover associations between random variables, and these dependencies can be used to perform prediction on future observations. Statistics has a long tradition in dealing with co-occurring events. For example, in medical statistics, scientists might want to find abnormal cell morphology from red blood cells that co-occur with a disease. Discovering this dependence can lead to the prediction of the occurrence of the disease based on a blood test. Built on statistical inference, machine learning exploits the vast computational power of modern computing platforms and develops efficient algorithms to solve complex real-world problems. During the last decade, machine learning has made spectacular progress, approaching or even surpassing human performance in complex tasks such as visual recognition [He et al.2015, Szegedy et al.2015], speech recognition [Hannun et al.2014], and computer gaming [Silver et al.2016].

In many machine learning problems, it is sufficient to model the dependencies between random variables in a probability distribution if the underlying data generating system is stable. However, many studies in scientific research and decision making aim to answer questions involving prediction under interventions outside the data-generating system [Pearl2000,Spirtes et al.2001]. For example, epidemiologists may wonder whether life expectancy will change if they advise people to change their diets. To predict the consequences of interventions, we need to understand the underlying mechanisms that generate the data. Causal relationships among variables provide coarse descriptions of the mechanisms, at a level sufficient to predict the system's behavior under interventions [Pearl and others2009]. The causal relationships encode asymmetric information between variables: by intervening on the cause we can control its effect, but not vice versa.

There are three major problems in causal inference: 1) mathematical formulation of causality, 2) identify and estimate causal effects, and 3) causal structure discovery. Mathematical formulation and representation of causation are fundamental to automated causal inference. Neyman-Rubin causal model [Rubin2005], Granger causality model [Granger1969], and the structural causal model (SCM) [Pearl2000] are three well known causal models that formulate causality from different perspectives. In particular, the structural causal model unifies other models and provides a coherent mathematical foundation for causal analysis. Identifying and estimating the effects of interventions (or causal effects) given a causal graph and empirical data, is our final goal of causal inference [Tian and Pearl2002, Shpitser and Pearl2006, Jaber et al.2019]. Discovering the causal structure is of practical importance because the causal structure is not always known in real applications.

In this paper, I will be focusing on two essential problems that bridge causality and learning and investigate how they can benefit from each other. On the one hand, since causality can be considered as directional statistical dependencies, causal inference naturally builds on the statistical associations and thus, many modern machine learning techniques can be explored for causal analysis. One typical example would be causal discovery from observational data. This problem can be viewed as a model selection problem that aims to search for the optimal causal graph explaining the data, leading to score-based causal discovery methods [Chickering2002]. As such, we can develop more advanced causal discovery approaches by exploiting modern learning techniques such as kernel methods and reinforcement learning [Huang et al.2018,Zhu and Chen2019]. Along this direction, we have developed new approaches for causal discovery from time series by exploring identifiable unsupervised learning methods that provide theoretical guarantees of the uniqueness of the learned causal graph [Gong* et al.2015, Gong et al.2017, Geiger et al.2015, Huang et al.2019, Chenwei et al.2019].

On the other hand, since causal structure provides information about the distribution changing properties, it can be used as a fundamental tool to tackle a major challenge for machine learning: the capability of generalization to new distributions and prediction in nonstationary environment. Motivated by the 'independence' postulate, which states that Pcause and Peffect|cause change independently, we have performed a series of studies on domain adaptation from a causal perspective [Zhang et al.2015,Gong et al.2016,Gong et al.2018,Zhang et al.2020]. In particular, we use causal models to represent the relationship between the features and labels and consider possible situations where different modules of the causal model change with the domain. We show that causal structures provide a powerful tool to estimate how the distribution changes across domains and develop efficient algorithms for adaptive learning. For other types of learning problems that can benefit from causality, please refer to a recent paper on causality for machine learning [Sch¨olkopf2019].

In the following two sections, I will briefly introduce our representative research works on learning causal structures from low-resolution time series and causal adaptive learning.

2 Learning Causal Relations from Low-resolution Time Series

Traditional approaches to causal inference from time-series data rely on the assumption that cause precedes effect, and in particular, Granger causality [Granger1969] is a widely applied causal discovery method for time series. However, Granger causality may be very different from true causality when the measurements are obtained at a coarser timescale (data resolution) than the underlying causal time scale (causal resolution). Recovering causal relations at the causal resolution from the lower-resolution data is difficult due to the information loss in the missing observations.

For Granger causal analysis in the linear case, one fits the following VAR model [Sims1980] on the data:

where xt = (x1t, x2t, ..., xnt)ᵀ is the vector of the observed data, et = (e1t, ..., ent)ᵀ is the temporally and contemporaneously independent noise process, and A contains the temporal causal relations. Suppose that due to the low resolution of the data, there is an observation every k time steps. That is, the low-resolution observations ˜X = (˜x1, ˜x2, , ..., ˜xt) are (x1, x1+k, ..., x1+(t-1)k); here we have assumed that the first sampled point is x1. We then have

We denote by ⃗et+1 the noise term, i.e., ⃗et+1 = ∑k-1 l=0 Al e1+tk-l. We call (A, et, k) the representation of the kth order subsampled time series ˜xt. A is called the causal transition matrix. Then we provide solutions to the following two problems.

• Identifiability of causal relations from low-resolution data

Due to information loss in the low-resolution data, estimating causal relations suffers from the identifiability problem: under the commonly adopted Gaussianity assumption of the data, the solutions are generally not unique. This non-identifiability problem has existed for decades, and many researchers believed that identifying causal relations from low-resolution data is hopeless. We proved that, however, if the noise terms are nonGaussian, the underlying model for the high-resolution data is identifiable from low-resolution data under mild conditions [Gong* et al.2015,Gong et al.2017].

• Estimation of causal relations by unsupervised learning models

We proposed practical algorithms for causal discovery from low-resolution data by transforming our model into non-Gaussian latent variable models. Then we can develop efficient learning procedures by variational inference [Gong* et al.2015] or adversarial learning [Chenwei et al.2019].

3 Causal Adaptive Learning

A common assumption in standard supervised learning is that the training and test set follow the same distribution. However, this standard assumption is often violated in many realworld applications due to intrinsic nonstationarity of the environment or inevitable sample selection bias. We propose causal domain adaptation methods for adaptive learning under distribution changes by injecting causal knowledge in the learning process.

Unsupervised domain adaptation aims to construct the classifier for the target/test domain, on which only the features xτ = (xτk )mk=1 are available, by making proper use of the data in s source/training domains (x(i) , y(i) ) = (x(i) k , y(i) k )mi k=1 , i = 1, ..., s. Traditional graphical models provide a compact, yet flexible, way to decompose the joint distribution of as a product of simpler, lower-dimensional factors [Pearl1994,Koller and Friedman2009], as a consequence of conditional independence relations between the variables. In particular, if the conditional independence relations can be presented by a Directed Acyclic Graph (DAG), then the joint distribution can be factorized as the product of factors, each of which corresponds to the conditional distribution of a variable given its parents. For our purpose, we need encode not only conditional independence relations between the variables, but also whether the conditional distributions change across domains. To this end, we propose an augmented DAG as a flexible yet compact way to describe how a joint distribution changes across domains, assuming that the distributions in all domains can be represented by such a graph (in other words, this graph encodes the property of the mother distribution behind the domain-specific distributions). It is an augmented graph in the sense that it is over not only features Xi and Y , but also external latent variables θ.

Figure 1 gives an example of such a graph. Nodes in gray are in the Markov Blanket (MB) of Y . The θ variables are mutually independent, and take the same value across all data points within each domain and may take different values across domains. They indicate the property of distribution shift-for any variable with a θ variable directly into it, its conditional distribution given its parents (implied by the DAG over Xi and Y) depends on the corresponding θ variable, and hence may change across domains. In other words, the distributions across domains differ only in the values of the θ variables. Once their values are given, the domainspecific joint distribution is given by P(X, Y | θ), which can be factorized according to the augmented DAG. By making use of such graphical representations, we can formulate domain adaptation as an inference problem on graphical models and reconstruct the joint distribution P(X, Y ) in the target domain by transferring supervision information from source domains.

4 Conclusion

In this paper, I briefly introduced our recent research on causality and machine learning. On the one hand, we explore modern machine learning techniques to learn causal structures from observational data. On the other hand, we explore causal knowledge for adaptive learning in nonstationary environment.