SeMi-LoRA: Enhancing Low-Rank Adaptation via Separation and Mixing

1 Introduction

Adapting large language models (LLMs) to application scenarios usually presents a significant challenge, especially under the growing demands of model training, evaluation, and deployment [Guo et al., 2023; Chen et al., 2026; Jin et al., 2024]. Full fine-tuning (FFT) is often impractical due to its high resource cost. Parameter-Efficient FineTuning (PEFT) techniques have thus become a critical research area [Houlsby et al., 2019], offering efficient solutions by optimizing a small subset of parameters while maintaining performance.

Among various PEFT methods, LoRA (Low Rank Adaptation) has emerged as one of the most widely adopted solutions due to its high flexibility and scalability [Hu et al., 2022]. LoRA approximates the updated weights by adjusting a set of low-rank adapters while keeping the original model weights frozen, thereby effectively reducing the computational resources and memory overhead. Furthermore, unlike methods such as Adapter [Pfeiffer et al., 2021] and P-Tuning [Li and Liang, 2021], LoRA's update parameters can be merged into the original model weights, eliminating additional inference latency. However, its low-rank approximation may introduce performance gaps [Ren et al., 2024]. A straightforward way to address this is by increasing LoRA's rank, but this results in a linear increase in computation and memory costs.

To address this limitation, various solutions have been proposed. [Zadouri et al., 2024] introduce the LoRA-MoE concept, which utilizes the Mixing of Experts (MoE) framework to integrate multiple LoRA modules for more flexible parameter updates. However, LoRA-MoE involves computationally complex nonlinear functions, which prevent the new parameters from being directly merged into the model weights. This increases both the inference cost and the difficulty of deployment. VeRA [Kopiczko et al., 2024] shares high-rank matrices with lightweight scaling, but its restricted expressivity often demands more compute to match performance. Sparse approaches also target higher effective rank. MELoRA [Ren et al., 2024] builds a high-rank block-diagonal update by composing multiple mini low-rank adapters, and MoRA [Jiang et al., 2024] uses non-parametric operators with a shared highrank matrix. However, these designs structurally restrict ∆W, limiting global information flow and preventing full updates across W. Overall, existing methods tend to trade off mergeability, efficiency, or complete update support; Table 1 summarizes these limitations.

Our goal is to achieve efficient high-rank updates while preserving parameter mergeability and the capacity for complete updates. Inspired by Depthwise Separable Convolution [Howard et al., 2017], we propose a novel PEFT method: SeMi-LoRA. Depthwise separable convolution separates a standard convolution into depthwise and pointwise operations, reducing computational complexity while preserving strong representation capacity. Similarly, we separate feature mixing across channels and apply a two-stage mixing strategy to the separated features.

Specifically, SeMi-LoRA consists of four stages: Compress, Rank Mix, Channel Mix, and Decompress. We first split features into non-overlapping channels and apply per-channel low-rank compression. We then perform two linear mixers in the latent space: Rank Mix (intra-channel, along rank) and Channel Mix (inter-channel, at fixed rank indices), enabling both local and cross-channel interactions. Finally, Decompress projects features back to the original dimension, yielding a mergeable update that supports global modifications of W with only marginal overhead.

Overall, our main contributions are as follows:

• We propose a novel PEFT method, SeMi-LoRA, which achieves high-rank updates by separating and mixing LoRA matrices.
• We provide two complementary perspectives of SeMiLoRA from feature mixing and matrix multiplication, and show that both MELoRA and LoRA are special cases of SeMi-LoRA.
• We conduct extensive experiments on commonsense reasoning, math reasoning, natural language understanding, and generation. Code and additional implementation details are available for reproducibility 1 . SeMi-LoRA consistently outperforms strong PEFT baselines, improving over LoRA by +5.02% and +7.58% absolute on commonsense and math reasoning, averaged across benchmarks.

2 RelatedWork

Parameter-Efficient Fine-Tuning (PEFT). With the widespread adoption of LLMs, fine-tuning them often requires substantial resources, which has driven the rapid development of Parameter-Efficient Fine-Tuning (PEFT) techniques. The core idea of PEFT is to enhance fine-tuning efficiency by selectively adjusting certain parameters of the model or introducing a small number of trainable parameters, without significantly increasing memory and computational demands [Houlsby et al., 2019]. Early research includes methods such as adapter [Pfeiffer et al., 2021], prefix tuning [Li and Liang, 2021], and prompt tuning [Lester et al., 2021], however, these approaches may lead to issues such as non-mergeable incremental parameters and increased inference costs. Other methods improve efficiency through partial fine-tuning, such as BitFit [Ben Zaken et al., 2022], or through feature scaling, such as IA3 [Liu et al., 2022], and re-parameterization fine-tuning, such as LoRA [Hu et al., 2022]. Notably, LoRA has become an important method for fine-tuning large-scale language models (LLMs) due to its high parameter efficiency and mergeable incremental parameters.

LoRA Extensions and Variations. LoRA has become a focal point of research owing to its effectiveness in fine-tuning large language models while demanding fewer computational resources. Building on LoRA's potential, researchers have proposed various extensions and optimization methods to enhance its performance and applicability. LoRA+ [Hayou et al., 2024] refines learning rate strategies to improve training effectiveness. DoRA [Liu et al., 2024] improves learning capacity and stability by decomposing weights into magnitude and direction. Furthermore, ReLoRA [Lialin et al., 2023] aims to increase the rank of the final parameters by continuously integrating current-step LoRA updates into the LLM during training. Meanwhile, VeRA [Kopiczko et al., 2024] further improves parameter efficiency by freezing the LoRA matrix and introducing trainable scaling vectors. Methods like LoRA-MoE [Zadouri et al., 2024] and MELoRA [Ren et al., 2024], through multi-LoRA expert ensembles, further enhance LoRA's flexibility and robustness. Related multilingual LLM studies also explore efficient training and LoRA-based knowledge transfer [Sun et al., 2024; Dong et al., 2025].

1 https://github.com/thinkingmanyangyang/peft-code

3 Preliminary

LoRA utilizes low-rank decomposition techniques to reparameterize model weight updates, allowing LLMs to adapt to new tasks. Formally, given a pre-trained weight matrix W0 ∈ Rdout×din , the expression for the weight update can be written as:

where ∆W ∈ Rdout×din represents the low-rank weight updates. In LoRA, ∆W is constructed as the product of two low-rank matrices A ∈ Rr×din and B ∈ Rdout×r , where the rank r ≪ min(din, dout). During the training process, the original weight matrix W0 remains frozen, while A and B are optimized as trainable parameters. During inference, the low-rank weight update matrix can be merged into the original matrix to reduce latency in the inference process.

4 SeMi-LoRA

We propose SeMi-LoRA, a parameter-efficient fine-tuning (PEFT) framework that reformulates separation-and-mixing for low-rank weight updates to approximate high-rank updates while remaining fully linear and mergeable. The architecture decomposes the feature transformation into four linear stages: Compress, Rank Mix, Channel Mix, and Decompress. Specifically, input features are partitioned into c channels and projected into an r-dimensional latent subspace. Subsequently, the representation undergoes alternating intra-channel mixing (Rank Mix) and inter-channel mixing (Channel Mix), both stabilized by residual connections, before being projected back to the output space. Figure 1 illustrates the framework from both feature mixing and matrix multiplication perspectives.

4.1 Feature Mix View of SeMi-LoRA

We use column vectors. Let the input feature be x ∈ Rdin . SeMi-LoRA follows a split-transform-merge paradigm.

Stage 1: Compress

We reshape x into c non-overlapping channels:

where ηc×g reshapes x ∈ Rdin into xv = [xv1 , . . . , xvc ] ∈ Rc×g with g = din/c. Here c is the number of channels.

For each channel i ∈ {1, . . . , c}, we compress xvi ∈ Rg into an r-dimensional latent subspace using a mini compression matrix Ai ∈ Rr×g :

obtaining xu = [xu1 , . . . , xuc ] ∈ Rc×r . We refer to r as the rank dimension (latent dimension per channel).

Stage 2-3: Rank Mix and Channel Mix

To enable expressive yet efficient interactions in the latent space, we introduce two linear mixers with residual connections, inspired by the MLP-Mixer design [Tolstikhin et al., 2021]. Rank Mix operates within each channel along the rank dimension, while Channel Mix operates across channels at each fixed rank index (without rank-to-rank mixing).

RankMix (intra-channel). For each channel i, we apply a trainable matrix Ri ∈ Rr×r with a residual connection:

Let R = [R1, . . . ,Rc]. Stacking all channels yields xr = [xr1 , . . . , xrc ] ∈ Rc×r .

Channel Mix (inter-channel). Channel Mix exchanges information across channels while preserving rank indices. We transpose xr ∈ Rc×r to obtain xr′ ∈ Rr×c , where the t-th row xr′ t ∈ Rc corresponds to features at rank index t across all channels. We then mix channels independently for each rank index t:

where C = [C1, . . . ,Cr] and each Ct ∈ Rc×c . After stacking all rows, we obtain xc′ ∈ Rr×c and transpose it back to xc ∈ Rc×r .

Stage 4: Decompress

Finally, we map the mixed latent features back to the output dimension. For each channel i, we use a mini decompression matrix Bi ∈ Rk×r with k = dout/c:

and aggregate xo′ ∈ Rc×k into the output vector

where γ(·) concatenates (or stacks) the c channel outputs back to the original output space.

4.2 MatrixMultiplication View andMergeability

A key advantage of LoRA is mergeability: the learned update can be folded into the base weight, incurring no inference overhead. We now show that SeMi-LoRA preserves this property. Since all stages above are linear, the overall transformation is also linear and can be written explicitly as a product of structured matrices.

Define the block-diagonal compression and decompression matrices Am ∈ Rcr×din and Bm ∈ Rdout×cr as

Rank Mix is block-diagonal across channels:

The residual form in Eq. (4) corresponds to multiplying by (Rm + I), where I ∈ Rcr×cr is the identity matrix.

Channel Mix as a sparse matrix. In the feature view, Channel Mix transposes xr ∈ Rc×r to xr′ ∈ Rr×c and mixes channels within each rank index. This transpose only permutes the same cr latent entries; therefore Channel Mix can be represented as a structured sparse matrix Cm ∈ Rcr×cr :

Here C ∈ Rr×c×c is a tensor, and C:,i,j ∈ Rr stores the mixing weights from channel j to channel i at each rank index. Each (i, j) block is diagonal (no rank-to-rank mixing), matching Eq. (5). The residual form in Eq. (5) corresponds to multiplying by (Cm + I).

Putting everything together, the overall weight update of SeMi-LoRA can be written as

which is a linear operator and thus can be merged into the base weights. We now characterize its rank capacity. For any feasible parameters,

Since Am = diagci=1 Ai and Bm = diagci=1 Bi, the blockdiagonal structure gives Rank(Am) = ∑ci=1 Rank(Ai) and Rank(Bm) = ∑ci=1 Rank(Bi). When the mini adapters are full-rank in the low-dimensional subspace, i.e., Rank(Ai) = Rank(Bi) = r, we have Rank(Am) = Rank(Bm) = cr. The residual mixers (Rm + I) and (Cm + I) are square operators in the same cr-dimensional latent space; except for degenerate cancellations, they do not reduce this latent rank. Therefore,

This shows that the channel number c expands the rank capacity of the update while preserving the linear and mergeable structure of SeMi-LoRA.

4.3 Relations withMELoRA and LoRA

We relate SeMi-LoRA to LoRA variants via the expansion:

Equivalently, this expansion contains the block-diagonal term BmAm and the cross-mixing term BmCmRmAm, which introduces cross-block channel interactions. This expands the support of ∆W beyond block-diagonal and enables non-local (cross-channel) updates, while the overall mapping remains linear and thus mergeable. The leading termBmCmRmAm forms a non-zero c × c block matrix, allowing cross-channel blocks (i, j) to be updated instead of restricting ∆W to independent block-diagonal groups.

MELoRA. Setting Rm = 0 and Cm = 0 disables both mixers and reduces the update to

Even with a high rank, the block-diagonal structure isolates input channels, preventing cross-channel interactions. SeMiLoRA overcomes this by employing Cm and Rm for channel and subspace mixing. This enables non-zero off-diagonals, enhancing the expressivity of ∆W without sacrificing mergeability.

LoRA. Standard LoRA uses ∆W=BAwith A ∈ Rr×din and B ∈ Rdout×r . Let A = [A1, . . . ,Ac] (column-wise) and B = [B⊤1 , . . . ,B⊤c ]⊤ (row-wise), where Aj ∈ Rr×g , Bi ∈ Rk×r , g = din/c, and k = dout/c. Set Rm = 0 and choose C such that C:,i,i = 0 and diag(C:,i,j) = Ir for all i ̸= j. Then Bm(Cm +I)Am produces blocks (i, j) = BiAj , yielding ∆W = BA. Therefore, SeMi-LoRA exactly reduces to LoRA under this construction.

5 Experiment

We evaluated SeMi-LoRA across commonsense reasoning, mathematical reasoning, text classification, and text generation, covering over 20 benchmarks. Baselines include LoRA [Hu et al., 2022], PiSSA [Meng et al., 2024], DoRA [Liu et al., 2024], VeRA [Kopiczko et al., 2024], MoRA [Jiang et al., 2024], HiRA [Huang et al., 2025], and MELoRA [Ren et al., 2024].

5.1 Commonsense Reasoning

Experiment Setup: We evaluated PEFT methods for commonsense reasoning by fine-tuning LLaMA2-7B [Touvron et al., 2023] and LLaMA3-8B with SeMi-LoRA, LoRA, and other baselines on eight datasets. The datasets include BoolQ [Clark et al., 2019], PIQA [Bisk et al., 2020], SIQA [Sap et al., 2019], HellaSwag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2021], ARC-e, ARC-c [Clark et al., 2018], and OBQA [Mihaylov et al., 2018]. These tasks are all in the form of multiple-choice questions. We followed the experimental settings of [Hu et al., 2023; Liu et al., 2024], integrated the training datasets of all eight tasks during the training process, and evaluated the model performance on the independent test set of each task. The performance of all models is measured using accuracy (%), and the best model is selected based on the validation set loss. We followed the optimizer, scheduler, and training protocol used in the corresponding baseline papers.

Result: As summarized in Table 2, SeMi-LoRA achieves the best average performance among the compared PEFT methods on commonsense reasoning benchmarks. On LLaMA27B, SeMi-LoRA attains 82.63% average accuracy, consistently surpassing partial-update approaches (e.g., MELoRA and MoRA) as well as high-rank variants such as HiRA under comparable parameter budgets. These results support our central hypothesis that a complete update formulation is crucial for faithful weight approximation in parameter-constrained regimes. Moreover, SeMi-LoRA exhibits a markedly improved efficiency-performance trade-off: it matches standard LoRA while using only one quarter of the trainable parameters. This advantage carries over to larger backbones: on LLaMA3-8B, SeMi-LoRA achieves 87.55%, demonstrating strong scalability and robustness against competitive baselines.

These results are consistent with the rank analysis in Section 4.2, which shows that channels can increase adaptation capacity without changing the base rank.

5.2 GLUE Benchmark

Experiment Setup: We employed the RoBERTa-large model on the GLUE benchmark [Wang et al., 2018] to evaluate the effectiveness of our method on natural language understanding (NLU). GLUE is a benchmark that includes classification, similarity/paraphrase, and natural language inference tasks. Following VeRA [Kopiczko et al., 2024], we evaluated on six GLUE tasks: CoLA, SST-2, MRPC, STS-B, QNLI, and RTE. We adopted the same evaluation metrics as [Hu et al., 2022] to ensure a consistent comparison.

We performed five runs using different random seeds and reported the median, following the standard GLUE fine-tuning protocol used by the compared PEFT baselines.

Result: Table 3 reports the GLUE results. With 0.9M trainable parameters, SeMi-LoRA obtains the highest average score among the compared PEFT methods, outperforming LoRA, MELoRA, and HiRA. Moreover, with only 0.24M trainable parameters, SeMi-LoRA remains close to the strongest baselines while using substantially fewer trainable parameters, suggesting favorable parameter efficiency on natural language understanding tasks.

5.3 Math Reasoning

Experiment Setup: Low-rank update methods (such as LoRA) often exhibit shortcomings when faced with math reasoning tasks that require deeper knowledge and reasoning abilities. These tasks depend on the model's knowledge accumulation and complex logical reasoning capabilities. LoRA's limitations lie in its low-rank update mechanism, which makes it difficult to effectively learn and store new knowledge [Jiang et al., 2024]. To evaluate SeMi-LoRA on mathematical reasoning, we fine-tuned LLaMA2-7B, Mistral, and Gemma on GSM8K [Cobbe et al., 2021] and MetaMathQA [Yu et al., 2023], and reported exact-match accuracy on the corresponding test sets using the last checkpoint. All experiments followed the optimizer, scheduler, and training protocol of [Meng et al., 2024].

Result: Table 4 shows that SeMi-LoRA consistently improves mathematical reasoning across backbones. On LLaMA2-7B, SeMi-LoRA achieves the best GSM8K and MATH scores among the compared PEFT methods. The gains also transfer to Mistral and Gemma, where SeMi-LoRA improves over LoRA under both parameter-efficient and comparable-parameter settings.

5.4 Open-Domain Dialogue Generation (ConvAI2)

Experiment Setup: We evaluated SeMi-LoRA on the ConvAI2 dataset [Dinan et al., 2019], which comprises 17,878 multi-turn dialogues for training and 1,000 for testing. Each example includes persona profiles and the dialogue history. Following prior work [Song et al., 2021], we adopted the self-persona setting, where only the speaker's own persona is provided as context. Experiments were conducted with two backbones, LLaMA-2-7B and LLaMA-3-8B, and we compared SeMi-LoRA against strong PEFT baselines. For evaluation, we reported BLEU [Papineni et al., 2002] and BERTScore [Zhang et al., 2019] to assess generation quality and semantic similarity, and additionally included METEOR and ROUGE-L for a more comprehensive comparison. We followed the optimizer, scheduler, and training protocol used by [Huang et al., 2025].

Result: The results are summarized in Table 5. Overall, SeMiLoRA delivers strong performance with both backbones and remains competitive with all baselines under comparable parameter budgets. Relative to standard LoRA and its variants (e.g., MoRA and HiRA), SeMi-LoRA consistently improves generation-focused metrics, including BLEU, METEOR, and ROUGE-L. These gains highlight the effectiveness of SeMiLoRA for persona-grounded open-domain dialogue generation.

6 Analysis

6.1 Does SeMi-LoRA's High Rank Improve Model Fitting Ability?

To examine whether SeMi-LoRA improves fitting ability through higher-rank updates, we constructed a two-spiral dataset with 1500 samples and used a multi-layer MLP to fit the non-linear decision boundary. The weightsWof the MLP linear layers were frozen, and different PEFT methods were used to approximate the update matrix ∆W. We trained all methods for 200 epochs with learning rate 5e-4.

Figure 2 visualizes the fitting results. Comparing LoRA Rank 8, Rank 32, and Rank 64 shows that increasing the rank improves the fitting of non-linear data. However, MELoRA, despite its partial high-rank design, remains limited because its block-wise update restricts global interactions. In contrast, SeMi-LoRA with rank 8 × 8 produces a decision boundary close to LoRA Rank 64, indicating that its separation-andmixing design effectively improves high-rank update capacity.

6.2 How is SeMi-LoRA's SystemEfficiency?

In this section, we explore the practical performance of the SeMi-LoRA method with a focus on system efficiency, primarily examining its performance in terms of GPU memory consumption and training latency. We compare the resource consumption of SeMi-LoRA with LoRA, VeRA, MELoRA, and DoRA when fine-tuning LLaMA2-7B (using the Commonsense Reasoning dataset) on an Nvidia A100 GPU. We use the same setup as the commonsense reasoning experiments.

As shown in Figure 3a, SeMi-LoRA, LoRA, VeRA, and MELoRA exhibit similar GPU memory consumption, indicating that SeMi-LoRA's GPU memory requirements are comparable to those of low-rank methods such as LoRA, without adding extra burden. DoRA, due to its complex parameter learning strategy, has higher memory consumption.

Figure 3b compares the training time of different methods. SeMi-LoRA, LoRA, and MELoRA show similar training durations, indicating that SeMi-LoRA remains computationally efficient despite the additional mixing stages. VeRA incurs slightly higher training time, possibly due to its very high-rank auxiliary frozen matrix, while DoRA shows more obvious training latency. This efficiency comes from the small additional computation in the compact latent space. Moreover, our einsum-based Channel Mix avoids large-scale activation permutes/transposes, reducing extra memory movement.

In summary, SeMi-LoRA surpasses the strongest existing baseline on commonsense reasoning while retaining LoRAlike memory usage and training throughput; compared to DoRA, it cuts training time by 54.4%, highlighting SeMiLoRA as a practical and efficient high-rank extension of LoRA.

6.3 What's the Component Effect on SeMi-LoRA?

To investigate the impact of feature separation and mixing strategies (Channel Mix and Rank Mix) in SeMi-LoRA on performance, we conducted a series of ablation experiments. All variants used the same backbone and training protocol, isolating the effect of each component. The experimental results are shown in Table 6. After removing the Rank Mix (RMix) mechanism and the Channel Mix (C-Mix) mechanism, the model performance decreased. This indicates that Rank Mix and Channel Mix are indispensable components in SeMiLoRA. The experimental results show that after removing the Residual Connection (Res) structure, the model performance slightly decreased. This suggests that the Residual Connection structure may play a role in integrating the advantages of various mixing methods. We also conducted experiments to swap the order of Rank Mix and Channel Mix. The results show that the order of Rank Mix preceding Channel Mix yields relatively better performance, but overall the performance is comparable.

6.4 HowDoes Initialization Impact Model Performance?

In this section, to investigate the impact of different initialization strategies on SeMi-LoRA performance, we designed various schemes and evaluated them on the GLUE Benchmark. We followed LoRA's initialization strategy, where matrix A in the LoRA decomposition is initialized with the Kaiming uniform distribution and matrix B is initialized with the zero matrix. For the Rank Mix module specific to SeMi-LoRA, we compared the initialization effects of orthogonal initialization and the Kaiming uniform distribution. For the Channel Mix module, we compared the differences between the Kaiming uniform distribution and all-one initialization. The experimental results of Table 7 show that SeMi-LoRA achieves optimal performance when the Rank Mix is initialized with the Kaiming uniform distribution, and the Channel Mix is initialized with all ones.

7 Conclusion

We have presented SeMi-LoRA, a method for improving the performance and efficiency of parameter-efficient fine-tuning on complex tasks. The core idea of SeMi-LoRA lies in a separation-and-mixing strategy, including Rank Mix and Channel Mix, which enables information interaction across feature dimensions, realizing a high-rank approximation of the parameter increment matrix. SeMi-LoRA combines feature-mixing and matrix-multiplication views, and supports merging new parameters into the original model weights during inference, achieving zero additional inference overhead. Extensive experiments on reasoning, understanding, and generation tasks validate the effectiveness of SeMi-LoRA, while analyses explore its mechanisms and efficiency under different settings.

8 Limitations

SeMi-LoRA demonstrates strong performance but introduces the channel number c as an additional hyperparameter, which may require tuning for new tasks or backbones. Rank Mix and Channel Mix also slightly increase implementation complexity compared with standard LoRA, although empirical memory and latency overhead remains small. Future work will investigate automatic channel selection and adaptive rank-channel allocation.

Acknowledgments

The present research was supported by the National Key Research and Development Program of China (Grant No. 2024YFE0203000). We would like to thank the anonymous reviewers for their insightful comments.

Contribution Statement

Zhenfei Yang and Beiming Yu contributed equally to this work. Deyi Xiong is the corresponding author.