Parameter-Efficient Fine-Tuning with Circulant and Diagonal Vectors

1 Introduction

Large foundation models (LFMs) are widely utilized in various fields, including natural language processing [Paaß and Giesselbach, 2023], image recognition and generation [Li et al., 2024a], medical diagnosis [Li et al., 2024b], and autonomous driving [Chen et al., 2024]. [Devlin, 2018] have proposed the bidirectional transformer architecture that understands input data from left to right and right to left. It is trained to predict missing words given input context, and it has served as a foundation model that can be fine-tuned for many downstream tasks. Following the transformer architecture, generative pre-trained transformer (GPT) model by [Radford and Narasimhan, 2018] handles input data from left to right following a sequential prediction order. This mechanism turns out successful in many generation tasks such as text summary, question answering, etc.

Although LFMs learn extensive general knowledge during the pre-training phase, they still require extra adjustments in downstream applications to effectively fullfill the task. Finetuning is a typical approach to continue learning on given downstream data and update from pre-trained model parameters. While fine-tuning significantly reduces computational costs compared to training from scratch, existing fine-tuning methods still suffer from the huge complexity of LFMs. As the original model parameters are still kept and maintained during fine-tuning stage, this leaves limited space for the development of fine-tuning methods.

To address the challenge of fine-tuning LFMs, [Hu et al., 2021] have proposed low-rank adaptation (LoRA). This method is an efficient fine-tuning approach designed for LFMs, reducing the number of parameters required during fine-tuning by introducing low-rank matrices. The essential idea is assuming the weight change matrix with low rank structure, expressing it as the product of two low rank matrices and only training these two smaller matrices while keeping the original weights frozen. FourierFT proposed by [Gao et al., 2024a] assumes a sparse structure in fourier domain of the weight matrix updates ∆W. Although FourierFT reduces the number of training parameters, the computational and storage requirements of the model remain very high, particularly when dealing with LFMs. The two-dimensional Fourier transform used to restore ∆W contributes to most of its computation and storage complexity. As a result, the fine-tuning model continues to necessitate high-performance hardware support, including substantial GPU resources and memory, which may be challenging to achieve in practical applications.

[Huhtanen and Per¨am¨aki, 2015] has demonstrated that a general complex matrix X ∈ Cn×n can be factorized into the product of multiple circulant matrices and diagonal matrices, with total number of factors not exceeding 2n - 1. This decomposition method offers several advantages, particularly in terms of computational efficiency and storage optimization. The computation and storage of diagonal matrices can be efficiently managed using vector representations. Moreover, circulant matrices possess a unique structure that allows them to be diagonalized using the fast Fourier transform (FFT), significantly reducing the complexity of matrix operations and accelerating computation speed.

Inspired by previous works, we propose circulant and diagonal vector based fine-tuning (CDVFT), which is also a Fourier domain based method. Our method represents the weight change matrix ∆W with the product of interleaved circulant and diagonal matrices. This factorization simplifies the matrix calculation process and reduces storage requirements. Due to the unique properties of matrix product for circulant and diagonal matrices, the quadratic computation complexity now becomes loglinear. Different from FourierFT based on 2D FFT, our fine-tuning process avoids the restoration of the weight change ∆W and only takes 1D FFT operations. However, the product of interleaved circulant and diagonal matrices inherently forms only square matrices, limiting its generality. To address this issue, we propose partitioning the circulant matrix into blocks, enabling more efficient storage management while maintaining the accuracy standards required for large-scale models. As a result, CDVFT can achieve efficient storage and computation at the same time. We summarize our main contributions as following:

• We introduce CDVFT method that represents ∆W using the product of interleaved block circulant and diagonal matrices. These matrices have linear storage complexity as each of them can be determined by a single weight vector. In practice, we find only a few matrices and a small number of blocks is sufficient to perform fine-tuning.
• CDVFT avoids the restoration of weight change matrix and has loglinear computation complexity. The circulant matrix vector product can be transformed into 1D FFT, and diagonal matrix vector product is linear in nature. Thus, the overall computation complexity becomes loglinear.
• We evaluate our method on natural language understanding, image classification, and instruction and task adjustment. Experimental results show that our method achieves similar or even better results in terms of moder performance, number of training parameters and FLOPs. For example, for the RoBERTa base model, our method results in 51.81× FLOPs reduction compared to FourierFT and 5.33× trainable parameters saving compared to LoRA, while resulting in similar or even better accuracy.

2 RelatedWorks

Fine-tuning LFMs is a challenging problem due to the large model size and computation requirement. Although training LFMs from scratch is performed on cloud platforms like LLaMA model by [Touvron et al., 2023], fine-tuning is often limited to a specific task and a low-cost computing environment. Besides, fine-tuning runs on a much smaller dataset than the pre-training dataset for LFMs. Thus, fine-tuning process is expected to be cost-effective. The overall complexity should be small and affordable in practice.

Full fine-tuning is a classical approach training and updating all model parameters at the same time. However, it is difficult to perform full fine-tuning on LFMs given the huge computation and storage requirement. [Brown et al., 2020] find LFMs are able to generalize to new tasks with few-shot demonstrations as prompt, thereby saving the effort of training on parameters. [Li and Liang, 2021] argue that adding few-shot demonstrations is bounded by the input length constraint of current LFMs. Instead, they propose the prefix tuning method to train a parameter vector and prepend to input, which is expected to work as prompt in unlimited length.

Updating all model parameters is not desirable in practice, since each task needs to maintain a model. [Houlsby et al., 2019] propose the adapter method, where task dependent parameters are inserted to LFMs. Fine-tuning process only updates those new parameters, thereby each task effectively sharing pre-trained LFM parameters. [Mahabadi et al., 2021] further reduce the task dependent parameters amount by grouping adapters into a hyper network model such that the network can produce task specific parameters on the fly. [Sung et al., 2022] discover backpropagation process through LFMs takes a lot of memory and propose a ladder style adapter design that significantly saves memory consumption. Given that adapters bring in extra inference latency due to their new parameters, [Lei et al., 2023] believe different tasks have different needs for the shared LFM architecture. They decide to learn to skip computations in LFM for different adapters, resulting in a faster inference speed.

It can be noticed that adapter adds task dependent parameters and incurs inference delay. There are also studies working on mergeable adapters so that after fine-tuning they can be merged into LFM architecture without adding inference latency. The essential idea is setting adapter parameters in the same shape as LFM pre-trained parameters, and fine-tuning learns the change of weight parameters, i.e., ∆W. [Hu et al., 2021] develop LoRA technique that enforces low rank structure into the weight change matrix. Given that LoRA rank can be different for different tasks, [Zhang et al., 2023] decide to learn the rank setting by modifying singular values based on importance score function. Instead of directly learning on ∆W, [Gao et al., 2024a] propose FourierFT to learn sparse parameters in fourier domain and reconstruct the weight difference using 2D FFT operation. It turns out this method requires much less number of parameters, but its reconstruction needs more memory.

Following the parameter efficient fine-tuning (PEFT) discovery in fourier domain, it is important to look for a method involving matrix and efficient FFT operation. Circulant matrix is related to 1D FFT since circulant matrix vector product can be executed using 1D FFT to accelerate. There are some studies applying circulant matrix to compress neural networks, such as circulant convolution neural network by [Cheng et al., 2015] and circulant long short-memory by [Wang et al., 2018]. However, these works are lack of flexibility on increasing parameter amount and theoretical guarantee on dense matrix approximation. [Huhtanen and Per¨am¨aki, 2015] has demonstrated that a general complex matrix X ∈ Cn×n can be expressed as the product of interleaved circulant and diagonal matrices, with the number of factors not exceeding 2n - 1:

where for j ∈ {1, . . . , n}, A2j-1 and C2j are diagonal and circulant matrices, respectively. Thus, this decomposition theoretically can approximate any dense matrix, and it also enables control on parameter amount by setting number of factors.

3 Method

In this section, we introduce circulant and diagonal vector based fine-tuning (CDVFT) method, which is a mergeable adapter design similar to FourierFT. After fine-tuning, our trained circulant and diagonal vectors can be used to build circulant and diagonal matrices, which are further combined to reconstruct ∆W and merged into LFMs. However, most importantly, CDVFT does not need to recover ∆W during real fine-tuning process, since the reconstruction results in high computation and storage complexity. Instead, our method takes advantage of the fast matrix multiplication algorithm from circulant and diagonal matrices involving 1D FFT and element-wise product to achieve the goal.

The overall computation flow is illustrated in Fig.1. It can be seen that CDVFT only takes vector operations at each step, thereby significantly reducing the computation and storage complexity. Specifically, according to the findings by [Huhtanen and Per¨am¨aki, 2015] and the unique properties of circulant and diagonal matrix operations, CDVFT first initializes corresponding vectors to represent these matrices. It then directly performs multiple element-wise multiplication and 1D FFT on input x. Finally, it yields the output ∆h, which can be added to the output h from the original weight matrix W.

3.1 Forward Step

Let x ∈ Rd×1 be an input column vector. Assume weight change matrix ∆W ∈ Rd×d that can be decomposed into 2m - 1 factors with m ≤ d. Thus, there are m diagonal matrices and m-1 circulant matrices. For j ∈ {1, 2, . . . , m}, each diagonal matrix is defined by a vector a2j-1 ∈ Rd×1 , and each circulant matrix is defined by a vector c2j ∈ Rd×1 .

More specifically, they can be expressed as following:

where diag(·) and circ(·) construct a diagonal matrix and circulant matrix, respectively. Therefore, the weight change matrix can be written as:

where × is the inner product operation. The end-to-end computation flow then becomes:

where α is a hyper-parameter scalar as in LoRA [Hu et al., 2021], W ∈ Rd×d is the pre-trained weight matrix in given LFM and h′ is the new output after adding our CDVFT adapters. This can also be seen in Fig. 1.

We perform the computation from rightmost to leftmost, thereby avoiding the reconstruction of ∆W during finetuning process. Let y ∈ Rd×1 represent the intermediate calculation result from matrix vector multiplications. Thus, y2j-1 is the result from diagonal matrix vector multiplication, and y2j is the result from circulant matrix vector multiplication. Note that diagonal matrix vector product is equivalent to element wise product of a2j-1 and input vector:

where ⊙ means the element-wise product. The circulant matrix vector product can be transformed into 1D FFT operations:

where ei2π p d q is the constant term in the Fourier transform, i indicates the imaginary unit, and p is the frequency index of the transform. We use letter F to indicate vectors in fourier domain. F2j-1 and F2j represent the Fourier transform results of y2j-1 and the circulant matrix vector c2j, respectively. ˆF is the result of element wise multiplication of F2j-1 and F2j. In consequence, y2j is the result of inverse fast Fourier transform (IFFT) of ˆF.

3.2 Backward Step

Following current deep learning design, we provide the gradient calculation with respect to a2j-1 and c2j for all j. Denote the objective function (i.e., loss function) as L(·). The backpropagation follows the chain rule, and we can get:

The backpropagation through the circulant matrix consists of derivatives of one-dimensional Fourier transform [Cheng et al., 2015], which is easier to write with explicit expression of FFT as shown in Eq. (8):

Note that ˆc2j and ˆy2j-1 are shifted from existing c2j and y2j-1, respectively. According to [Cheng et al., 2015], the shift pattern is fixed, i.e., from (0, 1, . . . , d - 1) to (0, d - 1, . . . , 1).

However, we find that this derivation is not efficient due to the need of applying FFT on shifted vectors. When the input is a real vector for FFT, it can be easily proved that the FFT on the shifted vector is equal to the conjugate of the result of FFT on original vector. Therefore, we further improve the backward step as following:

where the conj(·) means taking the conjugate of input vector. In this way, we show that the backward step can reuse some results from the forward step. Since FFT operation is the major computation complexity, it can be noticed that forward step takes 2 FFT and 1 IFFT, while backward step takes 2 IFFT and 1 FFT. Overall, both forward and backward steps have similar computation complexity.

3.3 Block Partition

The product of interleaved circulant and diagonal matrices inherently forms only square matrices. As diagonal matrix is always a square matrix, to overcome the limitation, we decide to adopt the block-wise partitioning strategy for circulant matrices [Ding et al., 2017]. This approach ensures compatibility with non-square weight matrices while preserving computational efficiency and storage benefits, maintaining the accuracy requirements of large-scale models.

In essence, this method partitions a non-square matrix into multiple square submatrices of equal size. If a dimension is not evenly divisible by the block size, it is automatically padded through replication. Each resulting square submatrix corresponds to a circulant matrix. Formally, given a matrix C ∈ Rn×n , we partition it into blocks of size p, resulting in q1 = ⌈d1/p⌉ blocks along the d1-dimension and q2 = ⌈d2/p⌉ blocks along the d2-dimension. Consequently, the original matrix is decomposed into q1 × q2 smaller circulant matrices. During matrix multiplication, block-wise multiplication is applied, where corresponding submatrices are multiplied element-wise. This effectively transforms the non-square matrix multiplication into multiple independent square circulant matrix multiplications, allowing us to fully leverage the computational properties of circulant matrices. The block-wise matrix-vector multiplication can then be formulated as:

Param. (M)

0.3

0.2

0.1

0.0

0.3M

0.3M

LoRA

38M

VeRA

24K

FourierFT

Param.

FLOPs

27M

Ours

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

• (a) Adapter complexity on RoBERTa-base model.
• (b) Adapter complexity on ViT-base model.

Figure 2: Complexity Analysis of different adapters. FF method is not presented due to its high cost in both parameters and FLOPs. Our proposed block circulant adapters can balance between parameter amount and FLOPs.

Each block matrix Ci,j represents a submatrix of the partitioned matrix, where i and j are integers ranging from 0 to (q1 - 1) and 0 to (q2 - 1), respectively. The vector ci,j ∈ Rp×1 corresponds to each small circulant matrix.

It is worth mentioning that this method was specifically proposed to improve the generalization of our method. We only need to enable the partition for the first circulant matrix such that the rest matrices can always be square for maximum efficiency. However, even in scenarios where weight matrices are already square, this method also has the potential to further enhance model performance since there are more learnable parameters after partitioning. In our experiments with RoBERTa and ViT, all fine-tuned weight matrices were inherently square. Therefore, we set p = d = 768 to achieve maximum efficiency. For experiments with LLaMA model, there are non-square weight matrices that we need to set different partition sizes to fit the training.

3.4 Complexity Analysis

Assume that the number of layers to be fine-tuned is Lt and ∆W ∈ Rd×d (Priority is primarily given to cases where the weight matrix is square.)

Parameters. The number of parameters Θ to be trained for LoRA is given by |Θ|LoRA = 2 × d × Lt × r, where | · | means the cardinality. For VeRA, the total number of trainable parameters is |Θ|VeRA = (r + d) × Lt. However, in practice VeRA method can take large r to achieve good performance. For FourierFT, let number of spectral coefficients be n, and the total number of trainable parameters is

43K

1.4G

55K

FLOPs (G)

|Θ|FourierFT = n×Lt. For CDVFT(Ours), in the case of square weight matrices, the circulant matrix block size be p = d. Assuming that the total number of circulant matrices and diagonal matrices is 2m- 1, the total number of trainable parameters is |Θ|CDVFT = (2m - 1) × d × Lt Fig. 2 shows that Fourier domain based method, i.e., FourierFT and ours, require much less number of parameters than LoRA.

FLOPs. Fig.2 also analyzes the computational complexity. It is important to note that computation complexity of FourierFT is independent of its parameter amount n since it always use 2D FFT to reconstruct ∆W. The complexity of our CDVFT is related to total number of circulant matrices and diagonal matrices and the circulant matrix block size, i.e., 2m- 1 and p . The computational complexity of CDVFT is smaller than FourierFT. The main reason for the complexity difference is that in FourierFT, the computational complexity of the 2D FFT for computing ∆W is O(d2 log(d2 ))). In CDVFT, the complexity brought by element-wise product and 1D FFT is O(mdlog(d)), which significantly reduces the computational complexity while keeping similar number of training parameters. It can be seen that across all models, LoRA has the largest parameter amount, and FourierFT has the largest FLOPs. The complexity of the different models listed in Table 3 verifies the analysis.

The case of non-square matrices. When the weight matrices in the fine-tuning layer are non-square, a block-wise partitioning strategy is required to ensure proper fine-tuning. In this case, the number of parameters and FLOPs depend on the block size p of the circulant matrix. A smaller p results in more circulant matrix blocks, increasing the number of parameters required for representation and making the computation more complex. Therefore, in our experiments, we prioritize choosing a larger p. To validate our reasoning, we present results under different values of p in Table 3. Furthermore, our approach achieves significantly lower FLOPs than FourierFT and requires fewer trainable parameters than LoRA, striking an effective balance between computational efficiency and model compactness.

4 Experiments

In this section, we evaluate our CDVFT method across different domains, i.e., natural language understanding (NLU) and computer vision (CV): (1) fine-tune the RoBERTa model [Liu et al., 2019] on the General Language Understanding Evaluation (GLUE) dataset [Wang et al., 2019]; (2) fine-tune the vision transformer model [Dosovitskiy et al., 2021] for various image classification tasks across different domains; (3) finetuning the LLaMA2-7b model on the Alpaca and GSM8K datasets.

Our proposed CDVFT is also compared with different adapters: Traditional full fine-tuning (FF) updates all model parameters, achieving high accuracy but incurring significant computational costs. Low-Rank Adaptation (LoRA) [Hu et al., 2021] is a widely adopted fine-tuning technique for large models. It decomposes the weight matrices into low-rank components, significantly reducing the number of trainable parameters while maintaining performance. Vector-based Random Matrix Adaptation (VeRA) [Kopiczko et al., 2024]

extends LoRA by sharing the low-rank matrices across all layers and inserting two additional vectors after the decomposed matrices, further improving efficiency. LaMDA [Azizi et al., 2024] is another LoRA-based method that freezes the first projection matrix (PMA) during fine-tuning, gradually freezes the second projection matrix (PMB) in the early training stages, and introduces a low-rank square matrix between them. LaMDA++ [Azizi et al., 2024] further extends this approach by assigning different ranks to different finetuning layers. FourierFT [Gao et al., 2024b] is a state-of-theart fine-tuning method that transforms frequency-domain data into trainable weight matrices via the Fourier transform. By training only in the frequency domain, it effectively reduces the number of trainable parameters. Notably, our approach also leverages the Fourier domain, but with lower FLOPs. This is achieved by employing a more efficient FFT operation in our equation (8) rather than the 2D FFT computation used in FourierFT.

4.1 Natural Language Understanding

Models and Datasets. We evaluate CDVFT on the GLUE benchmark dataset, which consists of a diverse range of NLP tasks, each representing a specific type of language understanding task. These tasks include question answering, sentiment analysis, textual entailment, etc. Following the experiment setting as in [Gao et al., 2024a], fine-tuning process runs on following tasks: CoLA, Corpus of Linguistic Acceptability [Warstadt et al., 2019], which determines whether sentences adhere to grammatical rules; SST-2, Stanford Sentiment Treebank [Socher et al., 2013], which classifies the sentiment of sentences as positive or negative; MRPC, Microsoft Research Paraphrase Corpus [Dolan and Brockett, 2005], which assesses whether two sentences convey the same meaning; STS-B, Semantic Textual Similarity Benchmark [Cer et al., 2017], which measures the semantic similarity score between sentence pairs; QNLI, Question Natural Language Inference [Rajpurkar, 2016], which evaluates whether the second sentence correctly answers the question posed by the first; and RTE, Recognizing Textual Entailment [Dagan et al., 2005], which identifies whether there is an entailment relationship between sentence pairs, functioning as a binary classification task. RoBERTa base model [Liu et al., 2019] is a transformer based foundation model, which is widely used in natural language processing. It improves over existing under-trained BERT model [Devlin, 2018] while preserving the powerful attention mechanism. Thus, it is selected to serve as the foundation model for GLUE dataset.

Implementation Details. Our CDVFT uses a total of 3 factor matrices, i.e., m = 2. The block size of the circulant matrix is the same as the matrix size, i.e., p = d = 768, and no block is performed. It should be noted that only query and value weights in each transformer block are finetuned, which is also applied to LoRA, VeRA and FourierFT as in [Gao et al., 2024a]. All implementations are in PyTorch [Paszke et al., 2019]. It can be seen that the optimizer is AdamW [Loshchilov, 2017]. For each dataset, there are different learning rates for foundation model language heads, query and value weight matrices. The scaling value is the α as in Eq. (4). The batch size and maximum input sequence length is set the same for all datasets.

Results. Table 1 summarizes fine-tuning results of all methods. The median metric value with standard deviation is reported out of 5 runs of experiments for each finetuning method, where each run takes a different random seed. The best performance for each dataset is highlighted in bold. Overall, compared with LoRA and FourierFT, our CDVFT method achieves comparable or even better performance. Besides, according to Fig. 2, our CDVFT results in 5.33× less number of trainable parameters than LoRA and 51.89× less FLOPs than FourierFT while fine-tuning RoBERTa base model on GLUE dataset.

4.2 Image Classification

Models and Datasets. The experiment evaluates the performance of our CDVFT method in image classification tasks, utilizing the Vision Transformer (ViT) by [Dosovitskiy et al., 2021] as the foundation model. Following the setting in [Gao et al., 2024a], we fine-tune on several challenging image classification datasets, only two are listed here for observation. RESISC45 [Cheng et al., 2017] provides a diverse range of remote sensing images; CIFAR-100 [Krizhevsky et al., 2009] is classical datasets of tiny images in 100 categories.

Implementation details. We set m = 2 and p = d = 768 for fine-tuning ViT base model across all these datasets. The ranks of LoRA and VeRA are 16 and 256 respectively. The n of FourierFT is set to 3000. For all method, fine-tuning only runs the query and value weight matrices of ViT, which is the same as in [Gao et al., 2024a]. The learning rate is set differently for fine-tuning ViT heads and query and value weight matrices.

Results. Table 2 presents the performance results on two image classification datasets after fine-tuning the ViT base model. Our CDVFT method demonstrates significant efficiency, requiring 10.7× fewer parameters than LoRA and 51.9× fewer FLOPs than FourierFT, while achieving similar or even better classification accuracy. Additionally, when compared to the latest fine-tuning method, VeRA, our approach uses fewer parameters. Although our method incurs a higher number of FLOPs than VeRA, it is important to note that VeRA's accuracy is relatively lower. To achieve comparable results to our method, VeRA would need to increase its rank, which would result in a corresponding increase in FLOPs.

4.3 Instruction Tuning

Models and datasets. Instruction tuning is a training method that enables models to learn how to perform tasks based on natural language instructions. Its primary goal is to enhance the versatility of models in handling a wide range of tasks, allowing for improved understanding and execution of natural language commands. Compared to traditional supervised learning, instruction tuning emphasizes the model's broad adaptability to task requirements and can be applied in scenarios such as question-answering systems, multi-task learning, and natural language generation.

In this experiment, FourierFT and CDVFT are used to fine-tune LLaMA2-7B on the Alpaca dataset. Specifically, LLaMA2-7B is part of the LLaMA (Large Language Model Meta AI) series developed by Meta, featuring significant improvements in performance and scalability over the first generation. This model generates high-quality text while utilizing fewer computing resources. The Alpaca dataset is developed based on the Stanford Alpaca project and expanded from instruction data generated by OpenAI's GPT model, focusing on tasks related to natural language understanding and generation. The experiment generates answers to predefined questions from MT-Bench, evaluated using GPT-4. MT-Bench is a benchmark tool specifically designed to assess multi-task language models, quantifying their generalization ability, speed, and accuracy by comparing performance across multiple tasks.

Additionally, the GSM8K dataset is used for task-specific fine-tuning and evaluation of the model. GSM8K consists of approximately 8,000 math problem-solving instances, aimed at improving the model's ability to perform complex reasoning tasks. The dataset includes problem descriptions and detailed solutions, enabling the model to learn how to solve complex mathematical problems.

Implementation details. For FourierFT, the experiments follow previous work and adopt the configuration n = 1000. For LoRA, r = 8 on the Alpaca dataset and r = 64 on the GSM8K dataset. For VeRA and LaMDA, r is 1024 and 32 respectively. For CDVFT, the experiments still set m = 2, and the partition size p is set as large as possible to achieve the minimum storage and computational cost. Therefore, p is set to 4096 and 2048.

Following [Gao et al., 2024b], we apply block circulant fine-tuning on query and value weight matrices inside the attention layer of two RoBERTa models and the LLaMA2-7B model fine-tuned on the alpaca dataset. (Except VeRA finetune on all layers in MHSA and MLP) Following [Azizi et al., 2024], we fine-tune on the MHSA and FFN layers of LLaMA2-7B model on the GSM8K dataset. The classification head is fully fine-tuned.

We evaluate the fine-tuned model on the Alpaca dataset using MT-Bench [Zheng et al., 2023], with GPT-4 [gpt, 2023] subsequently assigning scores to the model's responses for 80 multi-turn questions on a scale of 10.

Results. Table 3 shows the performance of the fine-tuned LLaMA2-7b model on the Alpaca and GSM8K datasets. It can be noticed that increasing partition size results in decreasing parameter amount and FLOPs. However, the task score or accuracy does not always decrease with larger partition size. This may be caused by the over-parameterization of the large LLaMA model or the strong structure of the proposed method that may serve as a regularization. Compared with other adapters, our method can balance between parameters amount and FLOPs.

5 Conclusion

Motivated by the recent success in Fourier domain based fine-tuning method, this paper proposes the CDVFT method that also learns parameters in Fourier domain. In particular, our method results in both trainable parameters savings and FLOPs reduction when compared with existing methods. The downstream task performance of our fine-tuned model achieves similar performance and sometime even better results across both natural language understanding and computer vision applications. These results effectively demonstrate the promising potential of our method and also the Fourier domain based fine-tuning methods.

Acknowledgments

This work was financially supported by the National Key R&D Program of China (Grant No. 2024YFA1211400).

Contribution Statement

Siyu Liao and Zhongfeng Wang are co-corresponding authors.