Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain

1 Introduction

Transformers [Vaswani et al., 2017] have achieved great success in many fields, including computer vision [Touvron et al., 2021; Liu et al., 2021], time series analysis [Li et al., 2019; Wu et al., 2021; Zhou et al., 2021], natural language processing [Nangia and Bowman, 2018; Maas et al., 2011; Radford et al., 2019; Devlin et al., 2019], and many other works [Wang et al., 2020; Shin et al., 2024; Kim et al., 2024; Yu et al., 2024]. Many researchers agree that the selfattention mechanism plays a major role in the powerful performance of Transformers. The self-attention mechanism employs a dot-product operation to calculate the similarity between any two tokens of the input sequence, allowing all other tokens to be attended when updating one token.

Figure 1: Illustration of the vanilla self-attention in Transformers and graph signal processing: (a) shows the softmax operation after the dot-product of query and key vectors, followed by the multiplication with the value vector; (b) and (c) show graph signal processing in undirected and directed graphs, respectively. For undirected graphs, the signal is filtered in the eigenvalue domain, while for directed graphs, the signal is filtered in the singular value domain.

Linear Transformers approximate the self-attention. However, the self-attention requires quadratic complexity over the input length to calculate the cosine similarity between any two tokens. This makes the self-attention difficult to apply to inputs with long lengths. In order to process long sequences, therefore, reducing the complexity of self-attention has become a top-priority goal, leading to the proposal of approximating the self-attention with linear complexity [Zaheer et al., 2020; Beltagy et al., 2020; Wang et al., 2020; Katharopoulos et al., 2020; Choromanski et al., 2020; Shen et al., 2021; Xiong et al., 2021; Qin et al., 2022; Chen et al., 2023]. However, existing self-attention with linear complexity aims to create a matrix that is close to the original self-attention map ¯A.

The self-attention is a low-pass filter (see Thm. 1). The self-attention map is a matrix that represents the relationship between every pair of tokens as scores normalized to probability by softmax. Considering each token as a node and the attention score as an edge weight, self-attention is considered as a normalized adjacency matrix from the perspective of the graph [Shi et al., 2022; Wang et al., 2022; Choi et al., 2024]. Therefore, the self-attention plays an important role in the message passing scheme by determining which nodes to exchange information with. Given that the self-attention operates as a normalized adjacency matrix, it is intuitively aligned with graph signal processing (GSP) [Ortega et al., 2018; Marques et al., 2020; Chien et al., 2021; Defferrard et al., 2016], which employs graph structures to analyze and process signals. Fig. 1 illustrates the selfattention in Transformers and signal filtering in GSP. As depicted in Fig. 1 (a), the original self-attention takes the softmax of the output from the dot product of the query vector and the key vector, and then multiplies it by the value vector. Fig. 1 (b) illustrates a general GSP method that applies the graph Fourier transform to the signal, conducts filtering in the spectral domain, and subsequently restores it to the original signal domain. In GSP, signals are filtered through the graph filters, which are generally approximated by a matrix polynomial expansion. The self-attention mechanism in Transformers can be viewed as a graph filter H defined with only the first order of the polynomial matrix, i.e., H " ¯A. Furthermore, since the self-attention is normalized by softmax and functions as a low-pass filter (see Theorem. 1), the high-frequency information in the value vector is attenuated, preventing the effective leverage of various frequency information. Consequently, existing self-attention mechanisms are designed in a rather simplified manner.

Our proposed linear Transformer learns an advanced self-attention (see Thm. 2). Although the approximation of linear Transformers is successful, what they are doing is simply a low-pass filtering. Therefore, to increase the expressive power of linear Transformers, we propose a more generalized GSP-based self-attention, called Attentive Graph Filter (AGF). We interpret the value vector of Transformers as a signal and redesign the self-attention as a graph filter. However, the existing self-attention mechanism possesses two problems: i) since the self-attention is based on directed graphs, the graph Fourier transform through eigendecomposition is not always guaranteed, and ii) the attention map changes for every batch, making it too costly to perform the graph Fourier transform every batch. In order to address the first problem, therefore, we design a self-attention layer based on the GSP process in the singular value domain (see Fig. 1 (c)). The singular value decomposition (SVD) has been used recently for the GSP in directed graphs and can substitute the eigendecomposition [Maskey et al., 2023]. In order to address the second problem, we directly learn the singular values and vectors instead of explicitly decomposing the self-attention map or any matrix. Since our proposed self-attention layer directly learns in the singular value domain by generating singular vectors and values using a neural network, our proposed method has a linear complexity of Opnd2 q. Therefore, our method efficiently handles inputs with long sequences. Our contributions can be summarized as follows:

1. We propose an advanced self-attention mechanism based on the perspective of signal processing on directed graphs, called Attentive Graph Filter (AGF), motivated that the self-attention is a simple graph filter and acts as a low pass filter in the singular value domain.
2. AGF learns a sophisticated graph filter directly in the singular value domainwith linear complexityw.r.t. input length, which incorporates both low and high-frequency information from hidden representations.
3. The experimental results for time series, long sequence modeling and image domains demonstrate that AGF outperforms existing linear Transformers.
4. As a side contribution, we conduct additional experiments to show that AGF effectively mitigates the oversmoothing problem in deep Transformer models, where the hidden representations of tokens to become indistinguishable from one another.

2 Background

2.1 Self-attention in Transformer

A key operation of Transformers is the self-attention which allows them to learn the relationship among tokens. The selfattention mechanism, denoted as SA: Rnˆd Ñ Rnˆd , can be expressed as follows:

where X P Rnˆd is the input feature and sA P Rnˆn is the self-attention matrix. Wq P Rdˆd , Wk P Rdˆd , and Wv P Rdˆd are the query, key, and value trainable parameters, respectively, and d is the dimension of token. The selfattention effectively learns the interactions of all token pairs and has shown reliable performance in various tasks.

However, in the case of the existing self-attention, a dotproduct is used to calculate the attention score for all token pairs. To construct the self-attention matrix sA P Rnˆn , the matrix multiplication with query and key parameters mainly causes a quadratic complexity of Opn2dq. Therefore, it is not suitable if the length of the input sequence is large. This is one of the major computational bottlenecks in Transformers. For instance, BERT [Devlin et al., 2019], one of the stateof-the-art Large Language Model (LLM), needs 16 TPUs for pre-training and 64 TPUs with large models.

2.2 Linear Transformer

To overcome the quadratic computational complexity of the self-attention, efficient Transformer variants have been proposed in recent years. Recent research focuses on reducing the complexity of the self-attention in two streams. The first research line is to replace the softmax operation in the self-attention with other operations. For simplicity, we denote softmaxpXWqrypXWkeyq⊺ q as softmaxpQK⊺ q. [Wang et al., 2020] introduce projection layers that map value and key vectors to low dimensions. [Katharopoulos et al., 2020] interprets softmax as kernel function and replace the similarity function with elupxq ` 1. [Choromanski et al., 2020] approximates the self-attention matrix with random features. [Shen et al., 2021] decomposes the softmaxpQK⊺ q into softmaxpQqsoftmaxpK⊺ q, which allows to calculate softmaxpK⊺ qV first, reducing the complexity fromOpn2dq to Opnd2 q. [Qin et al., 2022] replaces softmax with a linear operator and adopts a cosine-based distance reweighting mechanism. [Xiong et al., 2021] adopts Nystr¨om method by down sampling the queries and keys in the attention matrix. [Chen et al., 2023] employs an asymmetric kernel SVD motivated by low-rank property of the self-attention. However, these approaches sacrifice the performance to directly reduce quadratic complexity to linear complexity.

The second research line is to introduce sparsity in the selfattention. [Zaheer et al., 2020] introduces a sparse attention mechanism optimized for long document processing, combining local, random, and global attention to reduce computational complexity while maintaining performance. [Kitaev et al., 2020] use locality-sensitive hashing and reversible feed forward network for sparse approximation, while requiring to re-implement the gradient back propagation. [Beltagy et al., 2020] employ the self-attention on both a local context and a global context to introduce sparsity. [Zeng et al., 2021] take a Bernoulli sampling attention mechanism based on locality sensitive hashing. However, since they do not directly reduce the complexity to linear, they also suffer a large performance degradation, while having only limited additional speed-up.

2.3 Graph Convolutional Filter

The graph signal processing (GSP) can be considered as a generalized concept of the discrete signal processing (DSP). In the definition of DSP, the discrete signal with length n is represented by the vector x P Rn . Then for the signal filter g P Rn that transforms x, the convolution operation x ˚ g is defined as follows:

where the index i indicates the i-th element of each vector. GSP extends DSP to signal samples indexed by nodes of arbitrary graphs. Then we define the shift-invariant graph convolution filters H with a polynomial of graph shift operator S as follows:

where K is the maximum order of polynomial and wk P r´8,8s is a coefficient. The graph filter is parameterized as the truncated expansion with the order of K. The most commonly used graph shift operators in GSP are adjacency and Laplacian matrices. Note that Eq. (3) applies to any directed or undirected adjacency matrix [Ortega et al., 2018; Marques et al., 2020]. However, Eq. (3) requires non-trivial matrix power computation. Therefore, we rely on SVD to use the more efficient way in Eq. (5).

3 ProposedMethod

3.1 Self-attention as a graph filter

The self-attention learns the relationship among all token pairs. From a graph perspective, each token can be interpreted as a graph node and each self-attention score as an edge weight. Therefore, self-attention produces a special case of the normalized adjacency matrix [Shi et al., 2022; Wang et al., 2022] and can be analyzed from the perspective of graph signal processing (GSP). In GSP, the low-/highfrequency components of a signal x are defined using the Discrete Fourier Transform (DFT) F and its inverse F ´1 . Let ¯x " Fx denote the spectrum of x. Then, ¯xlfc P Cc contains the c lowest-frequency components of ¯x, and ¯xhfc P Cn´c contains the remaining higher-frequency components. The low-frequency components (LFC) of x are given as LFCrxs " F ´1 p¯xlfcq P Rn , and the high-frequency components (HFC) are defined as HFCrxs " F ´1 p¯xhfcq P Rn . Here, the DFT F projects x into the frequency domain, and F ´1 reconstructs x from its spectrum. The Fourier basis fj " re2πipj´1q¨0 , e2πipj´1q¨1 , . . . , e2πipj´1qpn´1q s⊺ {?n is used in computing F, where j denotes the j-th row. In GSP, the adjacency matrix functions as a low-pass filter, using the edge weights to aggregate information from nodes attenuates the high-frequency information of the nodes. In other words, the self-attention also acts as a low-pass filter within Transformers, and it is theoretically demonstrated below.

Theorem 1 (Self-attention is a low-pass filter). Let M " softmaxpZq for any matrix Z P Rnˆn . Then M inherently acts as a low pass filter. For all x P RN , in other words, limtÑ8}HFCrMt pxqs}2{}LFCrMt pxqs}2 " 0

The proof of Theorem 1 is in [Wi et al., 2025, Appendix D]. As the self-attention is normalized by softmax, the selfattention functions as a low-pass filter. Hence, Transformers are unable to sufficiently leverage a various scale of frequency information, which reduces the expressive power of Transformers.

Inspired by this observation, we redesign a graph filterbased self-attention from the perspective of GSP. As mentioned earlier, since the adjacency matrix can serve as a graph-shift operator, it is reasonable to interpret the selfattention as a graph-shift operator, S " ¯A. Moreover, the self-attention block of the Transformer is equivalent to defining a simple graph filter H " ¯A and applying GSP to the value vector, treated as a signal. Therefore, in Eq. (3), we can design a more complex graph filter through the polynomial expansion of the self-attention.

Note that when we interpret the self-attention ¯A as a graph, it has the following characteristics: i) The self-attention is an asymmetric directed graph, and ii) all nodes in the graph are connected to each other since the self-attention calculates the relationships among the tokens. Then we can derive that the self-attention is a special case of the symmetrically normalized adjacency (SNA) as ¯A " D´1A where A is an adjacency matrix and D is a degree matrix of nodes. In particular, SNA is one of the most popular forms for directed GSP [Maskey et al., 2023].

3.2 Polynomial graph filter

When approximating the graph filter with a matrix polynomial, k´1 matrix multiplications are required to calculate up to the k-th order (cf. Eq. (3)), which requires a large computational cost. Therefore, to reduce the computational complexity, we avoid matrix multiplications by directly learning the graph filter in the spectral domain. In the case of an undirected graph, a filter can be learned in the spectral domain by performing the graph Fourier transform through eigendecomposition. In general, however, the eigendecomposition is not guaranteed for directed graphs. The GSP through SVD, therefore, is often used [Maskey et al., 2023] if i) a directed graph sA is SNA and ii) its singular values are non-negative and within the unit circle, i.e., }sA} ď 1. For sA and its SVD sA " UΣV⊺ , an α-power of the symmetrically normalized adjacency is defined as:

where α P R [Maskey et al., 2023]. Therefore, we can define the graph filter H as follows:

where θ P Rn is a vector for singular value coefficients. Therefore, a spectral filter can be defined as a truncated expansion with K-th order polynomials. In other words, unlike directly performing the matrix polynomial as in Eq. (3), the computational cost is significantly reduced by K times element-wise multiplying of the singular values, which are represented as a diagonal matrix.

However, the polynomial expansion in Eq. (5) is parameterized with monomial basis, which is unstable in terms of its convergence since the set of bases is non-orthogonal. Therefore, for stable convergence, a filter can be designed using an orthogonal basis. Note that we have the flexibility to apply any basis when using the polynomial expansion for learning graph filters. In this work, we adopt the Jacobi expansion [Askey and Wilson, 1985], one of the most commonly used polynomial bases. Furthermore, Jacobi basis is a generalized form of classical polynomial bases such as Chebyshev [Defferrard et al., 2016] and Legendre [McCarthy et al., 1993], offering strong expressiveness in the graph filter design. Detailed formulas are provided in [Wi et al., 2025, Appendix F]. Therefore, we can define the graph polynomial filter as follows:

where Tkp¨q is a specific polynomial basis of order k.

3.3 Attentive Graph Filter

In order to use Eq. (6), however, we need to decompose the adjacency matrix sA, which incurs non-trivial computation. Therefore, we propose to directly learn a graph filter in the singular value domain (instead of learning an adjacency matrix, i.e., a self-attention matrix, and decomposing it). Therefore, as shown in Fig. 2, we propose our attentive graph filter (AGF) as follows:

whereWU,WΣ,WV P Rdˆd are learnable matrices, ρ is a softmax, and σ is a sigmoid. Our proposed model does not apply SVD directly on the computed self-attention or other matrices. Instead, the learnable singular values σpXWΣq and orthogonally regularized singular vectors UpXq and V pXq are generated by neural network. The singular values are then filtered by the graph filter, denoted as ΣpXq. To ensure that the elements of the singular value matrix are non-negative and within the unit circle, the sigmoid function is applied to the matrix. Moreover, we observe that the softmax of singular vectors enhances the stability of learning.

We construct our graph filter using the generated singular values, leveraging the Jacobi expansion as an orthogonal polynomial basis. If the trainable coefficients θk is allowed to take negative values and learned adaptively, the graph filter can pass low/high-frequency components of the value vector. Therefore, AGF functions as a graph filter that leverages various frequency information from the value vector. Furthermore, unlike the adjacency matrix that remains unchanged in GCNs, the self-attention matrix changes with each batch. To enhance the capacity for addressing these dynamics, AGF incorporates a token-specific graph filter, characterized by n different sets of singular values. This allows to leverage the token-specific frequency information in the singular value domain, increasing the capability to handle complex dynamics in hidden representation.

3.4 Objective Function

In the definition of SVD, UpXq is column orthogonal, V pXq is row orthogonal, and ΣpXq is a rectangular diagonal matrix with non-negative real numbers. When we train the proposed model, strictly constraining UpXq and V pXq to be orthogonal requires a high computational cost. Instead, we add a regularization on them since these matrices generated by neural network can be trained to be orthogonal as follows:

where I P Rdˆd is an identity matrix. Therefore, our joint learning objective L is as follows:

where Ltransformer is an original objective function for Transformers. The hyperparameters γ controls the trade-off between the loss and the regularization.

3.5 Time and Space Complexities of AGF

Since our AGF is based on the concept of SVD, it is not restricted by softmax for calculating attention scores. Therefore, UpXq, ΣpXq, and V pXq generated by neural network can be freely multiplied according to the combination law of matrix multiplication. First, since ΣpXq is a diagonal matrix, by performing element-wise multiplication with UpXq and the diagonal elements of ΣpXq, pn ˆ dq matrix is calculated with a time complexity of Opndq. Next, by multiplying V pXq and the value vector, pdˆdq matrix is calculated with a time complexity of Opnd2 q. Finally, by multiplying the outputs of steps 1 and 2, the final output is pnˆdq matrix with a time complexity of Opnd2 q. Therefore, the time complexity is Opnd2 q and the space complexity is Opnd ` d2 q.

3.6 Properties of AGF

How to use high-frequency information. In GSP, the characteristics of the graph filter are determined by the learned coefficients θk of the signal. These coefficients allow the graph filter to function as a low-pass, high-pass, or combined-pass filter, depending on the specific needs of each task [Defferrard et al., 2016; Marques et al., 2020; Chien et al., 2021], demonstrated by following theorem:

Theorem 2 (Adapted from [Chien et al., 2021]). Assume that the graph G is connected. If θk ě 0 for @k P t0, 1, ...,Ku, řKk"0 θk " 1 and Dk1 ą 0 such that θk1 ą 0, then gθp¨q is a low-pass graph filter. Also, if θk " p´αqk , α P p0, 1q and K is large enough, then gθp¨q is a high-pass graph filter.

The proof is in [Wi et al., 2025, Appendix E]. Theorem 2 indicates that if the coefficient θk of a graph filter can have negative values, and learned adaptively, the graph filter will pass low and high frequency signals appropriately. This flexibility is crucial for effectively processing signals with varying frequency components. Similarly, AGF operates as a filter that modulates frequency information in the singular value domain through the generated singular values and singular vectors. This approach enables AGF to dynamically adjust the frequency components of the signal, providing a more tailored and efficient filtering process. Therefore, unlike conventional Transformers, AGF can appropriately incorporate both low and high frequencies for each task, thereby enhancing the expressive power and adaptability of Transformers.

Comparison with existing linear self-attention methods.

We explain that while our AGF addresses the computational inefficiencies inherent in the vanilla self-attention like existing linear self-attention studies, we take a different approach from them. Instead of using explicit SVDs, our AGF reinterprets self-attention through a GSP lens, using the learnable SVD to learn graph filters directly from the spectral domain of directed graphs. Linformer [Wang et al., 2020], the most prominent representative of linear self-attention, approximates the vanilla self-attention through dimensionality reduction, and Nystr¨omformer [Xiong et al., 2021], which reduces to linear complexity with a kernel decomposition method, also efficiently approximates the full self-attention matrix with the Nystr¨om method. Singularformer [Wu et al., 2023], a closely related approach, uses a parameterized SVD and linearize the calculation of self-attention. However, like existing linear Transformers, it approximates the original self-attention, which is inherently a low-pass filter. Thus, to the best of our knowledge, existing linear self-attention methods focus on approximating the self-attention and reducing it to linear complexity, whereas our AGF approximates a graph filter rather than the self-attention. This allows AGF to use the token-specific graph filter to improve model representation within the singular value domain.

4 Experiments

4.1 Time Series Classification

Experimental settings. To evaluate the performance of AGF, we employ UEA Time Series Classification Archive [Bagnall et al., 2018] which is the benchmark on temporal sequences. Strictly following [Wu et al., 2022], we report accuracy for 10 multivariate datasets preprocessed according to [Zerveas et al., 2021]. We adopt 2-layer Transformer as backbone with 512 hidden dimension on 8 heads and 64 embedding dimension of self-attention. The experiments are conducted on 1 GPU of NVIDIA RTX 3090. The detailed descriptions are in [Wi et al., 2025, Appendix H.1].

Experimental results. Table 1 summarizes the test accuracy of AGF and the state-of-the-art linear Transformer models on the UEA time series classification task. We observe that AGF achieves an average accuracy of 75.1, outperforming the vanilla Transformer and other linear Transformers by large margins across various datasets. This performance gap underscores the effectiveness of our approach in leveraging advanced graph filter-based self-attention to enhance the expressive power of Transformers.

4.2 Long Range Arena Benchmark

Experimental settings. We evaluate AGF on Long Range Arena (LRA) [Tay et al., 2020] benchmark under longsequence scenarios. Following [Xiong et al., 2021], we train 2 layer Transformer with 128 hidden dimension, 2 heads, and 64 embedding dimension with mean pooling. The experiments are conducted on 1 GPU of NVIDIA RTX 3090. The details are in [Wi et al., 2025, Appendix H.2].

Experimental results. We report the top-1 test accuracy on LRA benchmark in Table 2. Our model demonstrates the highest average performance, achieving a score of 60.1 - an improvement of 1.3 points over the vanilla Transformer. In contrast, SingularFormer, a close approach that parameterizes SVD, only functions as a low-pass filter and thus fails to achieve optimal performance. Compared with YOSOE, a state-of-the-art linear-complexity Transformer, AGF improves the performance by a substantial margin.

4.3 Sensitivity Analyses

We conduct sensitivity studies on K and γ. Other sensitivity studies are reported in [Wi et al., 2025, Appendix K].

Sensitivity study on K. We test our model by varying K on UEA time series classification, and the results are shown in Table 3. As K increases, the performance improves. However, beyond a certain threshold, increasing K results in saturation and diminished performance. Therefore, choosing an appropriate K has a significant impact on performance.

Sensitivity study on γ. Table 4 summarizes the impact of γ on LRA benchmark. The optimal level of regularization applied to learnable singular vectors varies depending on the dataset, and we demonstrate that imposing a certain degree of regularization can enhance training stability.

4.4 Empirical Runtime

Table 5 summarize the results of the runtime and peak memory usage during the training phase. AGF consistently improves the efficiency of both time and space complexity compared to the vanilla Transformer. Specifically, for Text dataset, which have extremely long input sequences, the efficiency of AGF stands out even more. When compared with other linear complexity Transformers, our AGF shows comparable efficiency with longer sequences.

4.5 Ablation Studies

We conduct various ablation studies, and the additional results on Tp¨q and ρ are in [Wi et al., 2025, Appendix L].

Effect on the graph filter. To analyze the impact of graph filter, we conduct an ablation study on the following variants: i) HUV ⊺ refers to the graph filters with parameterized singular vectors and the singular values are fixed as one; ii)

HSV D initializes the singular values to one, allowing them to be learnable fromHUV ⊺; and iii) AGF refers to the proposed method. Table 6 shows the result of the effect of the graph filter, and in general, these ablation models leads to suboptimal performance. However, AGF processes the generated signal through the graph filter, allowing the model to use various scales of frequency information. The graph filter enhances the capacity of the model, resulting in optimal performance and demonstrating the effectiveness of AGF.

4.6 Additional Experiments on Deep Transformer

Experimental settings. We conduct additional experiments for image classification task with ImageNet-100 [Russakovsky et al., 2015] and ImageNet-1K [Deng et al., 2009] datasets and report top-1 accuracy. We adopt DeiT-small as the backbone, and trained from scratch with 300 epochs [Touvron et al., 2021] with 2 GPU of NVIDIA RTX 3090. The detailed descriptions are in [Wi et al., 2025, Appendix H.3].

Experimental results. Table 7 shows the top-1 accuracy on ImageNet-100 and ImageNet-1k. Our AGF effectively learns the representation in deep layers model, which has 12 layers. Notably, plugging AGF improves the performance marginally, from 80.6 to 81.3 trained on ImageNet100 datasets and from 79.8 to 80.3 on ImageNet-1K.

Analysis on mitigating over-smoothing problem. Deep Transformers, like GCNs, suffers from over-smoothing problem, where hidden representations become similar and indistinguishable to the last layer [Kipf and Welling, 2017; Veliˇckovi´c et al., 2018; Oono and Suzuki, 2020; Rusch et al., 2023]. We previously demonstrated that the selfattention in Transformers acts as a low-pass filter attenuating high-frequency information, which is a major cause of over-smoothing [Wang et al., 2022; Shi et al., 2022; Choi et al., 2024]. AGF mitigates this issue by effectively leveraging various scale frequency information through directly filtering signals in the singular value domain. Fig. 3 (a) illustrates the frequency information in both the vanilla DeiT (i.e., H " ¯A) and DeiT + AGF (i.e., H " UpXqΣpXqVpXq⊺ ). Unlike the vanilla model, AGF better captures high-frequency information. Additionally, Fig. 3 (b) shows the cosine similarity among hidden vectors at each layer. While the cosine similarity in DeiT increases to nearly 0.9 as layers deepen, it is moderated to nearly 0.5 in DeiT + AGF. Thus, AGF prevents over-smoothing in deep Transformers by effectively leveraging diverse frequency information.

5 Conclusions

We presented AGF, which interprets the self-attention as learning graph filters in the singular value domain from the perspective of directed graph signal processing. Since the self-attention matrix can be interpreted as a directed graph, we designed a more expressive self-attention using signals directly in the singular value domain. By learning the coefficients for various polynomial bases, AGF uses diverse frequencies. Our experiments showed that AGF outperforms baselines across various tasks, and the training time and GPU usage of AGF are comparable to baseline models with linear complexity. As a side contribution, AGF mitigates the oversmoothing problem in deep Transformers.

Since our comparison scope is focused on linear Transformers, a limitation is the exploration and comparison regarding the recent state-space models [Gu et al., 2021; Gu and Dao, 2023]. Exploring the potential of our method in enhancing state-space models is an intriguing avenue for future work.

Acknowledgments

This work was partly supported by Institute for Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. RS-2022-II220113, Developing a Sustainable Collaborative Multi-modal Lifelong Learning Framework, and 80% No. RS-2024-00457882, AI Research Hub Project, 10%), and Samsung Electronics Co., Ltd. (No. G01240136, KAIST Semiconductor Research Fund (2nd), 10%)