Tensor Network: from the Perspective of AI4Science and Science4AI

1 Motivation and Contribution

Tensor networks (TNs) have become a powerful computational tool for efficiently representing and processing highdimensional data structures, playing a crucial role in addressing the growing complexity of problems both in artificial intelligence (AI) [Wall and D'Aguanno, 2021; Liang et al., 2024; Su et al., 2024] and scientific research [Bachmayr et al., 2016; Sch¨utt et al., 2017; Or´us, 2019]. TN is essentially a network of tensors, which are high-dimensional arrays, connected by contractions along shared indices. Each tensor can be visualized as a node, and the indices or legs represent the dimensions along which tensors are connected or contracted. The power of TNs lies in their ability to decompose and represent large data structures with potentially exponentially fewer parameters compared to what would be required without such structured decomposition [Bridgeman and Chubb, 2017].

∗Correspondence author.

The inception of TNs is rooted in the realm of quantum physics, where they were originally developed as tools to solve the many-body quantum systems problem [Or´us, 2019; Yuan et al., 2021]. Their ability to factorize high-dim tensors into more manageable lower-dim components allows them to become indispensable in science especially physics. This factorization inherently resolves the curse of dimensionality, converting an exponential set of variables into a polynomial scale. Consequently, TNs have transcended their traditional boundaries and have become instrumental in general AI, from data science [Oseledets, 2011; Lu et al., 2021] to machine learning [Stoudenmire and Schwab, 2016; Xiong et al., 2024], and more recently in the design and optimization of AI models [Lu et al., 2021; Xiong et al., 2025].

In AI, TNs enable the efficient handling of high-order correlations in data, a task where traditional linear methods may fall short. They have been successfully integrated into the training processes of neural networks [Kossaifi et al., 2020; Panahi et al., 2020], providing substantial reductions in parameter sizes through tensor decompositions-such as the Canonical Polyadic, Tucker, and Tensor Train decompositions-while preserving model expressiveness. This integration is not only computationally efficient but also enhances model scalability, as cited in recent studies that highlight reductions in operational complexities in deep learning architectures while maintaining or even enhancing accuracy [Kossaifi et al., 2020; Wang et al., 2023].

The utilization of TNs in scientific contexts is equally transformative. In fields like material science and quantum computing, they facilitate precise simulations of molecular interactions and quantum states, processes that are otherwise computationally prohibitive [Simeon and De Fabritiis, 2024]. For example, TNs allow for the simulation of quantum circuits through matrix product states and operators, enriching scientific insights with computational models that mirror physical behaviors [Phan et al., 2018].

The adoption of TNs in both AI and scientific research underscores a critical paradigm shift: the recognition that high-dimensional data and complex correlations can be systematically captured through structured and physics-inspired decompositions. However, this interdisciplinary subject has also exposed gaps in understanding how TNs fundamentally unify principles from science and AI. A key motivation of this survey lies in revealing a picture that empowers researchers to leverage TNs not merely as algorithmic tools but as conceptual bridges between disciplines. The paper highlights:

1. AI4Science: TNs synergize with deep learning to model high-dimensional quantum systems, enabling precise simulation of entanglement dynamics and material properties beyond classical computational limits.
2. Science4AI: TN-inspired tensor decompositions compress high-dimensional AI models, with the potential of reducing model's size while preserving expressivity and enhancing efficiency through multilinear algebraic priors.

2 Preliminaries on Tensor Network

Normal letters represent scalars (e.g., a). Bold lowercase letters represent vectors (a). Bold uppercase letters denote matrices (A). Calligraphic uppercase letters are used for tensors (T ), which generalize matrices to higher dimensions. The description of the notations are given in Tab. 1.

2.1 Tensor Operations and Notations

An element of a high-order tensor can be accessed by multiple indices. E.g., an element Tijk of a 3-D tensor T ∈ Rm×n×p can be accessed by the indices i ∈ {1, 2, . . . , m}, j ∈ {1, 2, . . . , n} and k ∈ {1, 2, . . . , p}.

Tensor Product: It generalizes the outer product of vectors. For two tensors A ∈ RI1×I2 and B ∈ RJ1×J2 , the outer product produces a higher-order tensor C ∈ RI1×I2×J1×J2 with elements:

Tensor product is essential in TNs, combining tensors into higher-dimensional structures.

Mode-n Multiplication: It is a generalization of matrix multiplication. Given a tensor T ∈ RI1×···×IN and a matrix A ∈ RJn×In , mode-n multiplication is defined as:

This is crucial in tensor decompositions like Tucker decomposition, enabling the manipulation of high-dimensional data.

2.2 Tensor Decomposition

Tensor decomposition techniques factorize a high-order tensor into a product of lower-order tensors, similar to singular value decomposition (SVD). These decompositions help approximate tensors with lower-rank components, reducing computational complexity.

Canonical Polyadic (CP) Decomposition: It expresses a tensor as a sum of rank-1 tensors. For a 3-D tensor T ∈ RI×J×K , the CP decomposition is:

where ar ∈ RI , br ∈ RJ , and cr ∈ RK , and R is the rank of the decomposition, i.e., the number of components in the low-rank approximation.

Tucker Decomposition: Ass a more flexible tensor factorization method, it decomposes a tensor into a core tensor and factor matrices. For 3D T ∈ RI×J×K , its decomposition is:

where G ∈ RR1×R2×R3 is the core tensor. A ∈ RI×R1 , B ∈ RJ×R2 , and C ∈ RK×R3 factor matrices.

Tensor Train (TT) Decomposition: The Tensor Train decomposition approximates high-dimensional tensors by a chain of low-rank tensors. For a 3D tensor T ∈ RI×J×K , the TT decomposition is expressed as:

where Ar1 ∈ RI×r1 , Br1,r2 ∈ Rr1×J×r2 , and Cr2 ∈ Rr2×K are the factor tensors.

3 Tensor Network for Science &Engineering

TNs have been an effective tool for science, especially in quantum mechanics related areas, e.g., chemistry, quantum many-body physics, quantum field theory etc. In this section, we primarily focus on the applications of tensor networks in the efficient simulation of quantum many-body systems and the calculation of the ground state. We in general take an AI4Science perspective in this section.

3.1 MPS and PEPS in Quantum Systems

TNs have become essential for simulating quantum manybody systems as they efficiently represent complex quantum states by decomposing them into smaller, manageable tensors. These networks are especially powerful when dealing with larger many-body systems, where traditional methods like exact diagonalization or full wave function representations are computationally prohibitive. Matrix Product States (MPS, Fig. 1a) and Projected Entangled Pair States (PEPS, Fig. 1c) are most widely used among the various types of TNs, as they provide powerful tools for simulating quantum systems, particularly in low-dimensional settings.

MPS is a subclass of TNs that represent quantum states on a one-dimensional lattice. The quantum state is represented as a product of matrices associated with each lattice site, with their contraction yielding the wave function. MPS is particularly useful for one-dimensional systems or systems with weak entanglement, where the state can be efficiently approximated using relatively low bond dimensions. Specifically, an MPS representation of a quantum state |ψ⟩

on a chain of L sites for open boundary conditions takes the form [Schollw¨ock, 2011]:

Here, Aσi are tensors of dimension D×D×d, with d the dimension of the local Hilbert space and D the bond dimension governing the amount of entanglement captured. MPS have proven highly efficient for simulating one-dimensional systems, as they allow for a compact representation of quantum states with polynomial scaling in L and D.

PEPS, on the other hand, is a more general class of TNs suited for two-dimensional systems. PEPS represents quantum states on a two-dimensional lattice and extends the MPS by using tensors that have additional indices corresponding to the two-dimensional grid. These tensors are connected in a way that mimics the structure of quantum entanglement across a 2D lattice [Or´us, 2014]. A PEPS representation of a quantum state on a lattice of L × L sites takes the following form:

Each tensor Aσi,j ai,j at position (i, j) is a d × D × D × D tensor (where D is the bond dimension), and the network structure allows for an efficient representation of the wavefunction. PEPS are particularly well-suited for representing strongly correlated systems, such as those occurring in higher-dimensional lattice systems, and for capturing complex patterns of entanglement.

While MPS is highly efficient for one-dim systems, PEPS provides a natural extension for higher-dim systems, where the entanglement is generally more complex and cannot be simply captured by MPS representation. Both MPS and PEPS are part of the broader TN framework, which allows for the efficient representation and manipulation of quantum states. Besides, neural network-based models have been investigated recently in representing and simulating quantum systems and demonstrate superior performance in tasks such as tomography [Quek et al., 2021], properties prediction [Tang et al., 2024; Tang et al., 2025] and quantum sensing [Cimini et al., 2021]. Furthermore, hybrid models that integrate the physical priors of TNs with the expressive power of neural networks have been proposed [Chen et al., 2023], offering a promising avenue for simulation of complex quantum systems.

3.2 TNs for Ground State Calculations

One of the primary applications of MPS and PEPS is the calculation of the ground state of quantum many-body systems. The general approach to finding the ground state using MPS or PEPS involves expressing the Hamiltonian in the corresponding TN formalism (MPO for MPS and PEPO for PEPS), and then minimizing the energy expectation value through variational optimization.

For MPS, the ground state |ψ⟩ is approximated by iteratively optimizing the tensors Aσi at each lattice site. The ground state energy is minimized by adjusting the tensors to lower the expectation value E = ⟨ψ| ̂H|ψ⟩ ⟨ψ|ψ⟩ . The primary technique used in this optimization is the Density Matrix Renormalization Group (DMRG) method, which has proven to be highly efficient for one-dimensional systems [Schollw¨ock, 2011]. For example, for the antiferromagnetic Heisenberg chain, the systemHamiltonian ̂H is:

The Hamiltonian can also be represented by sums of tensor products of operators, ̂H = J z ̂ Sz 1 ⊗ ̂ Sz 2 ⊗ ̂ I ⊗ ̂ I ⊗ ̂ I · · ·+ ̂ I ⊗ J z ̂ Sz 2 ⊗ ̂ Sz 3 ⊗ ̂ I ⊗ ̂ I · · · + · · · , which directly corresponds to Matrix Product Operators (MPO, Fig. 1b).

DMRG optimizes the MPS by adjusting the tensors at each site through alternating sweeps over the system. At each site, a local eigenvalue problem of the tensor is solved, and the MPS is updated using Singular Value Decomposition (SVD) to maintain the desired bond dimension. This iterative optimization continues until the energy converges to its minimum. DMRG is particularly efficient for onedimensional systems and can handle large systems with hundreds of sites. However, its applicability is relatively limited for systems with long-range interactions or in higher dimensions, where the entanglement grows more rapidly and MPS may not efficiently capture the full complexity of the quantum state [Markov and Shi, 2008].

For PEPS, the Hamiltonian is expressed as a Projected Entangled Pair Operator (PEPO, Fig. 1d), which is a generalization of the MPO for two-dimensional systems. The PEPO facilitates the calculation of the energy expectation value ⟨ψ| ̂H|ψ⟩ through the contraction of the PEPO and PEPS tensors [Or´us, 2014]. Similar to MPS, the PEPS tensors are optimized iteratively to minimize the energy. The optimization process involves adjusting the tensors at each lattice site while keeping the others fixed, alternating between different sweeps over the lattice to ensure convergence to the ground state.

4 Tensor Network for Machine Learning

TNs have become a powerful tool in machine learning, offering improvements in both efficiency and expressiveness. This chapter focuses on two main applications: compressing neural network weights and parameter-efficient fine-tuning, and using tensor networks for expressive data representation that captures local and global correlations in tasks like network embedding and multimodal fusion. In this section, we try to take a Science4AI perspective.

4.1 TN forWeightedMatrix Compressing

In neural networks, the trainable parameters are typically represented as matrices or higher-order tensors with considerable size, which introduce significant computational and resource challenges. Moreover, as the model becomes more complex, there is a higher risk of overfitting, which can negatively impact the model's ability to generalize to unseen data.

Recent research has shown that generalization can be improved by restricting or refining the degrees of freedom within these high-dimensional tensors. Shallow tensor network methods, such as MPS [Perez-Garcia et al., 2007] and MPO [Pirvu et al., 2010], offer an elegant solution by representing these large tensors more compactly, achieving remarkable compression rates without sacrificing performance. The underlying principle is that a dense weight matrix is redundant for accurate linear transformations in neural networks due to the predominantly short-range correlations in data. For instance, in a fully connected layer, the rank of the weight matrix is inherently constrained due to short-range correlations or entanglements among input pixels [Denil et al., 2013]. Consequently, a lower-rank matrix can efficiently substitute the original without loss of predictive performance.

Many works have studied the efficiency of tensorizing the components of neural networks to bridge low-order parameter spaces to high-dimensional representations. For example, the MPO representation can significantly reduce the number of variational parameters, as its parameter count scales linearly with system size. [Gao et al., 2020] proposed to use MPO to represent linear transformation matrices in deep neural networks, and their results show MPO cannot only improve the efficiency in training and reduce the memory space, but also slightly improve the test accuracy using much fewer number of parameters than in the original networks. Specifically, they first reshape the weight matrix W into a 2n-indexed tensor Wj1...jn,i1...in . The MPO representation of W is obtained by factorizing it into a product of n local tensors Tr (w(1) [j1, i1]w(2) [j2, i2] · · ·w(n) [jn, in]), where w(k) [jk, ik] is a Dk-1 × Dk matrix. The tensor elements of w(k) , instead of the elements of W, are the variational parameters in the training procedure of deep neural networks. The number of parameters increases with the increase of the bond dimension Dk, which serves as a tunable parameter that controls the expressive power.

[Qing et al., 2023] proposed automatically differentiable tensor network (ADTN), which encodes the variational parameters into the contraction of tensor networks that contains exponentially fewer parameters, and utilizes automatic differentiation technique to reach the optimal accuracy after compression. The experiments showed that they can compress two linear layers in VGG-16 with approximately 107 parameters to two ADTN's with just 424 parameters, where the testing accuracy on CIFAR-10 is improved from 90.17% to 91.74%. [Xie et al., 2024] leveraged the emerging tensor ring (TR) factorization to compress the neural network and proposed an approach based on prime factorization that simultaneously identifies the optimal tensor reshaping and TR decomposition. To identify the optimal execution order, they construct a novel tree structure to schedule the execution of core tensors, minimizing computational complexity.

Pre-trained large language models (LLMs) have revolutionized natural language processing (NLP) across various tasks. However, as model sizes continue to grow, full finetuning becomes impractical due to its high computational and memory costs. To address this challenge, parameter-efficient fine-tuning (PEFT) techniques have been developed, allowing model adaptation by updating only a small subset of parameters. Low-Rank Adaptation (LoRA) [Hu et al., 2021] is a classic PEFT methods, which fine-tunes LLMs by introducing low-rank matrices into the pre-trained model's weight updates. Recently, some research has extended LoRA using tensor-based techniques [Bershatsky et al., 2024] to reduce the number of trainable parameters within the low-rank framework. Considering that low-rank approximations may sometimes result in a performance gap compared to full finetuning, especially for complex tasks, QuanTA [Chen et al., 2024] introduced a novel efficient high-rank fine-tuning inspired by quantum circuit structures, surpassing the limitations of LoRA. However, the hyperparameters in QuanTA, such as the number of tensors applying on the same axes, have not been optimized. Therefore, choosing an optimal set of tensors could further enhance their performance.

The requirement for precise tensor-rank selection and effective decomposition strategy poses computational and algorithmic hurdles, often leading to NP-hard problem. The permutation of tensor entries and the selection of optimal tensor-rank configurations can impact the efficacy drastically, necessitating efficient heuristic approaches [Li et al., 2022]. Despite these challenges, recent works in TN structure search algorithms, e.g. Alternating Local Enumeration (TnALE) [Li et al., 2023], offer promising pathways to avoid combinatorial explosion and prove exponentially convergence speedup compared to previous study [Li et al., 2022].

4.2 TN for Expressive Representation

TNs also provide a mathematically grounded framework for efficient representation learning, inspired by the tensor product of n quantum states, which spans a 2n -dimensional Hilbert space. By strategically composing low-dimensional sub-tensors, these models achieve exponential parameter compression while preserving high-dimensional semantic relationships. This dual capability to capture both local interactions and global correlations makes them highly expressive.

In natural language processing, the word2ket [Panahi et al., 2020] method decomposes word embeddings into rankr order-n entangled tensors using the formulation:

It reduces the storage complexity of traditional d×p embedding matrices from O(dp) to O(rnq log p/q). It has shown significant space savings in text summarization tasks.

Extending node2ket [Xiong et al., 2024] to the setting of network embedding, the framework constructs node representations as product states across orthogonal subspaces:

enabling multi-scale relationship modeling through hierarchical inner products:

where ⟨·, ·⟩ denotes the inner product operation. This architecture has been successfully applied to large-scale networks, achieving high accuracy in tasks such as network reconstruction and node classification.

Emerging extensions integrate TNs with deep architectures, notably through quantum-inspired dynamic rank adaptation [Hua et al., 2022] and hybrid tensor-transformer designs [Ma et al., 2019], enabling TNs as a unifying paradigm for high-dimensional representation learning.

TNs with their multilinear structure and efficient data representation, can capture complex relationships within data while reducing computational complexity. Compared to traditional DNNs, TNs show promise in tasks like multimodal data fusion [Baltrusaitis et al., 2019]. Integrating TNs could improve the performance of deep learning models and address some of their limitations.

TNs have proven to be a powerful tool for efficiently representing and fusing high-dimensional data. Zadeh et al. [Zadeh et al., 2017] introduced TNs with deep information fusion layers, named Tensor Fusion Layers (TFLs), for multimodal sentiment analysis. TFLs were designed to capture both intra-modality and inter-modality dynamics, effectively aggregating multimodal interactions, which was crucial for tasks like multimodal sentiment analysis involving text, visual, and acoustic data.

TFL begins by embedding the three vectors zt, zv, and za for each modality, rather than use raw data types. It then concatenates a scalar 1 with each of these embedded vectors:

Next, it computes the outer product of these concatenated vectors to form a feature tensor Z:

This tensor is subsequently processed by a two-layer fully connected neural network to produce a prediction. The TFL captures both unimodal and multimodal interactions, making it more expressive than simple concatenation-based fusion, which only models unimodal interactions.

However, while TFLs offer strong fusion capabilities, their computational complexity increases exponentially as the number of modalities increases, which can lead to inefficient training. To address this, Low-Rank Multimodal Fusion (LMF) [Liu et al., 2018] was introduced to reduce the computational burden and overfitting risk. LMF employs a special BTT (block term decomposition) layer, reducing the complexity of the TFL from O (∏Mm=1 dm) to O (dh ∑Mm=1 dm), where dm represents the dimensions of each modality, and dh is the hidden dimension.

Though TFL and LMF show improved fusion, they restrict the order of interactions, which leads to a loss of higher-order interaction information. To tackle this issue, the Polynomial Tensor Product (PTP) [Hou et al., 2019] block is proposed. PTP merges all feature vectors into a long feature vector:

It then creates a polynomial feature tensor of degree P as:

This tensor is processed by a tensor layer, such as a CP (canonical polyadic) layer, to model high-order interactions:

where Wi are weight matrices and Λ is a learnable diagonal matrix. The PTP block, similar to deep polynomial neural networks, captures all nonlinear and high-order interactions.

In addition to multimodal data fusion, TNs can also capture high-order relationships within each modality. For example, in image-sentence matching tasks, TNs help overcome the limitations of traditional methods, which typically match global features in a shared space, neglecting intra-modality dynamics. The Cross-Modal Hybrid Feature Fusion (CMHF) framework [Xu et al., 2021] directly learns image-sentence similarity by fusing multimodal features with both inter- and intra-modality relations. By using flexible attention mechanisms, TNs capture fine-grained interactions within each modality, improving the overall similarity learning process. Therefore, even outside of multimodal tasks, TNs are still highly effective in general machine learning. In these cases, TNs can be applied in a similar manner to decompose and reassemble feature vectors at each layer, capturing higher-order interactions between the data.

5 Tensor Network Development and Tools

Computational techniques and tools essential for efficiently handling TNs in modern applications. This section discusses efficient algorithms for tensor operations, parallel and distributed computing with GPUs, and key software for TNs. Additionally, we highlight advancements in compression and resource management, focusing on adaptive techniques for optimizing efficiency, memory usage, and accuracy.

5.1 Efficient Algorithms for Tensor Operations

Tensor contraction and decomposition are helpful for mitigating computational overhead while maintaining data integrity in high-dimensional space. The challenge lies in optimizing the contraction sequence to minimize cost. Strategies like contraction search heuristics and dynamic programming have proven effective, but further optimization for large-scale settings is needed [Taylor, 2024]. The choice of contraction path significantly impacts performance, with recent advances focusing on hybrid strategies combining heuristic and deterministic methods to improve efficiency [Li et al., 2022].

Decomposition techniques like CP, Tucker, and Tensor Train decompositions simplify complex tensor structures, boosting efficiency. These methods offer different trade-offs between accuracy and computational cost. CP is simple and interpretable but suffers from high complexity, particularly for higher-order tensors [Phan et al., 2018]. Tucker provides more flexibility by allowing multi-dimensional cores, at the cost of increased computational expense. TT handles largescale tensors by approximating them through low-rank tensor contractions [Kossaifi et al., 2020].

Emerging methods, such as Hierarchical Tucker (HT) decomposition, use hierarchical partitioning to decompose tensors into smaller, more manageable components, aiding in both data compression and feature preservation [Cichocki et al., 2017]. By recursively breaking down high-dimensional data into lower-dimensional sub-tensors, HT efficiently captures multi-level correlations within the data. This hierarchical structure significantly alleviates the curse of dimensionality, which typically leads to exponential growth in parameter space for high-dimensional datasets. As a result, HT becomes particularly advantageous for scalable applications where computational efficiency is critical.

Current research emphasizes integrating TNs with machine learning frameworks, where tensor operations optimize model sizes and computational loads [Yang et al., 2023]. This intersection opens new avenues for applying tensorization, such as tuning neural network architectures to enhance performance and reduce overhead [Wang et al., 2023].

As the field progresses, it is crucial to develop algorithms that adaptively manage resources and leverage hardware advancements while prioritizing efficiency in complex tensor operations. Future research should explore innovative methods like quantum-inspired computations and adaptive learning strategies [Rieser et al., 2023]. These efforts will help overcome computational barriers and broaden tensor network applications across AI and computational science.

5.2 Parallel &Distributed Computing Techniques

Central to parallel and distributed computing is effective partitioning of computations across multiple processors while minimizing communication overhead. Embedding tensor contractions in GPUs has accelerated computations by optimizing kernel operations and reducing data transfer overhead between CPU and GPU memory [Schutski et al., 2020], enabling manipulating larger tensors than would be possible sequentially. To further enhance scalability, distributed tensor computations allocate workloads across multiple processors. Frameworks like MPI enable inter-node communication, allowing tensor decomposition into smaller, concurrently processable chunks [Cichocki, 2014b]. This approach is advantageous for large-scale simulations and data analytics.

An emerging trend is dynamic load balancing and adaptive resource allocation, which ensures even workload distribution, minimizing idle times and optimizing resource usage. These strategies are crucial as TN applications expand into more heterogeneous and dynamic environments [Cichocki et al., 2017]. Additionally, cloud-based platforms increasingly support scalable resource allocation, allowing for elastic scaling to meet tensor computation demands, particularly in realtime data processing [Cichocki, 2014b].

As data size and dimensionality increases, maintaining consistency and synchronization across nodes can escalate overhead, reducing parallelism benefits. Designing algorithms to map tensor operations efficiently onto hardware, especially with irregular or sparse tensors, remains a significant challenge [Kanakagiri and Solomonik, 2024].

Innovative strategies include heuristic and probabilistic graphical models to optimize contraction paths without exhaustive searches [Schutski et al., 2020], reducing complexity and enhancing performance.

Looking ahead, integrating machine learning to predict optimal parallel execution configurations can further improve efficiency and scalability. Such models could dynamically adjust task partitioning and resource allocation based on computation patterns and data structures. Quantum computing principles may also offer novel solutions to scaling bottlenecks [Robeva and Seigal, 2019].

5.3 Software Tools and Libraries

Key tools include deep learning libraries such as TensorFlow and PyTorch, which have integrated TNs into their ecosystems. These platforms allow for the seamless expansion of traditional neural network architectures into tensor-inspired frameworks. TensorFlow's adaptable computational graph and PyTorch's dynamic computation graph support realtime execution, offering flexibility for researchers exploring TN algorithms in machine learning frameworks. These platforms are useful for developing high-dimensional models [Cichocki, 2014a].

Specialized tensor software like TensorLy offers powerful abstractions tailored for tensor operations. It supports a wide range of decompositions, including CPD and Tucker, and allows customization of decomposition processes to suit specific data properties, making it valuable for scientific applications [Rabanser et al., 2017].

ITensor focuses on quantum simulations and condensed matter physics, specializing in high-precision quantum TN computations. It treats tensor manipulations as accessible objects, in contrast to traditional linear algebra approaches, and uses a graphical representation of tensors to handle complex quantum states and operations [Cichocki et al., 2017].

TensorNetwork and SciNet blend NumPy's simplicity with TN contraction and optimization capabilities. Enhanced by efficient numerical schemes, these libraries support large tensor data and benefit from GPU acceleration for practical feasibility in scientific computing [Kolda and Hong, 2020].

Graphical tools like TensorTrace and GuiTeNet facilitate tensor visualization, aiding researchers in building, testing, and visualizing tensor models, thus reducing cognitive load and enhancing model interpretability [Cheng et al., 2020].

Emerging trends emphasize improving computational efficiency, resource management, and scalability. Integrating tensor compression and parallel computation techniques addresses the curse of dimensionality [Yin et al., 2021]. These libraries also support distributed computing and adaptive resource allocation for sustainable AI models that minimize energy consumption and promote efficient tensor handling.

Despite advancements, challenges remain, particularly in ensuring numerical stability when working with highdimensional tensors. Hybrid tensor architectures and on-thefly compression strategies continue to be key areas for research, aiming to develop more robust tools for various scientific and industrial applications [Wu et al., 2020].

5.4 TNCompression and Resource Management

TN compression uses techniques like CP, Tucker, and TT decompositions to reduce the storage and computational demands of high-dimensional data. These methods transform high-dimensional tensors into lower-dimensional components, offering substantial space savings [Cichocki et al., 2014; Rabanser et al., 2017].

TT decomposition is particularly notable for its linear storage costs and ability to preserve key properties of the decomposed components [Cichocki et al., 2017]. Its hierarchical structure and the Tensor Ring variant allow for flexible rank adjustments that optimize storage while maintaining data fidelity. These efficiencies are evident in quantum computing and machine learning, where complex data are handled effectively [Biamonte et al., 2017; Tang and Yan, 2022]. A critical aspect of compression is adaptive rank selection, where the ranks of decomposed components adjust dynamically to data complexity. This adaptability ensures efficient storage without compromising accuracy, facilitating the use of TNs on memory-limited hardware and improving performance on GPUs and high-performance platforms [Charlier et al., 2021].

Emerging trends highlight sparse tensor computing to optimize storage and performance. Sparse tensor can reduce storage needs without sacrificing computational accuracy. Compiler support systems like MLIR assist in generating optimized sparse tensor operations, easing developers' workload [Bik et al., 2022]. These innovations underscore the benefits of tensor compression and resource management.

However, balancing compression with precision remain challenging. Excessive compression can compromise essential data features, affecting model accuracy. Further exploration of methods to maintain precision while employing aggressive compression is essential [Ma et al., 2024]. Advancing adaptive ranking techniques and sparse tensor algorithms are critical for progress. Future research should focus on refining cost-based tensor program optimizers, allowing users to balance storage, computation speed, and resource usage. Studies could lead to new TN applications, especially in energy-efficient AI, where resource management is vital.

5.5 Evaluation of Computational Trade-Offs

A key trade-off in TNs is between computational precision and operational efficiency. High-precision tensor operations, necessary in quantum simulations and scientific computations, demand significant resources and can lead to long execution times. For example, the precision needed in quantum many-body simulations increases the complexity of tensor contractions and decompositions, requiring a careful balance [Ferris and Poulin, 2013].

Efficient use of computational resources is another key aspect of TN implementations. Memory footprint and compute time are integral to evaluating performance across different scales. Recent studies propose methods to improve resource allocation efficiency, reducing overhead without significant accuracy loss [Yin et al., 2021].

Effective resource management integrates memoryefficient algorithms and hardware acceleration. The application of GPUs and distributed frameworks helps mitigate computational constraints, leveraging parallel processing [Wu et al., 2024]. Benchmarking plays a vital role in evaluating TN algorithms. Standardized workflows and performance metrics are used to assess computational trade-offs and optimize contraction paths and network configurations [Ram´ırez et al., 2024].

Performance analysis tools like ITensor and TensorNetwork enable systematic comparisons of TN algorithms under various conditions [Fishman et al., 2022]. Despite progress, challenges remain, such as optimizing multi-way arrays and scaling across diverse platforms. Continuous research into novel decompositions, contraction strategies, and hardware advancements is needed to enhance the adaptability and effectiveness of TNs [Li et al., 2022].

6 Conclusion and Outlook

In this paper, we have explored the role of TNs in both AI and scientific research. We have demonstrated how TNs, initially developed for quantum many-body problems, have evolved to serve as powerful tools for addressing complex, highdimensional data in AI applications, ranging from deep learning to machine learning model optimization. In scientific computing, TNs facilitate efficient simulations of quantum systems, material properties, and molecular interactions, thus overcoming the limitations of traditional methods. Through their ability to decompose high-dimensional data into manageable components, TNs enable both significant computational savings and enhanced model expressivity.

Looking ahead, the potential for TNs to bridge AI and scientific disciplines is immense. In the context of AI4Science, TNs can be integrated with deep learning models to offer new insights into quantum systems and complex physical phenomena, with applications ranging from quantum chemistry to material science. On the other hand, Science4AI applications are likely to see a significant expansion, with TNinspired tensor decompositions offering ways to compress large-scale AI models, reducing their size while maintaining or improving performance. Furthermore, the combination of TNs with hybrid quantum-classical machine learning models may open new frontiers in both AI and quantum computing.

As these interdisciplinary developments progress, there is an increasing need for unified frameworks and open-source tools that facilitate the seamless application of TNs across domains. This paper provides a foundation for future research and offers an optimistic outlook for the further evolution of tensor networks in solving some of the most challenging problems in science and AI.

Acknowledgments

This work was in part supported by National Natural Science Foundation of China (92370201, 62222607, 623B1009).