LoCo: Low-Rank Compositional Rotation Fine-Tuning

1 Introduction

Large-scale pretrained foundation models [Vaswani et al., 2017] have revolutionized computer vision and natural language processing, achieving impressive performance across diverse downstream tasks. Models such as large language models [Brown et al., 2020], vision transformers [Dosovitskiy, 2020] and diffusion transformers [Rombach et al., 2022; Peebles and Xie, 2023] contain billions of parameters, encoding rich representations learned from massive

through full fine-tuning is often impractical: it demands substantial computational resources, risks catastrophic forgetting of pretrained knowledge, and requires storing separate model copies for each task. Parameter-efficient fine-tuning (PEFT) methods [Houlsby et al., 2019; Hu et al., 2022] have emerged as an effective solution for task adaptation while training only a small fraction of parameters.

Orthogonal fine-tuning methods [Qiu et al., 2023; Liu et al., 2024b; Yuan et al., 2024] have attracted attention for their geometric properties. Unlike additive methods such as LoRA [Hu et al., 2022] that injects low-rank adaptation to weight matrices, orthogonal methods apply multiplicative transformations, which are constrained to the special orthogonal group. This constraint offers a theoretical advantage in that orthogonal transformations preserve the pairwise angular relationships among neuron representations and maintain the hyperspherical energy structure of pretrained features [Qiu et al., 2023]. Such properties are particularly valuable when semantic information is encoded in angular relationships rather than magnitudes, as observed in contrastive learning [Chen et al., 2020] and face recognition [Liu et al., 2017].

Despite these properties, existing orthogonal fine-tuning methods face a fundamental computational bottleneck. Constructing and applying a general orthogonal matrix requires cubic time complexity for matrix inversion or matrix exponential computation, which becomes prohibitively expensive for high-dimensional feature spaces typical in modern architectures. Prior work addresses this challenge through structural constraints. For example, OFT [Qiu et al., 2023] uses block-diagonal matrices that restrict rotations to independent subspaces, while BOFT [Liu et al., 2024b] uses butterfly factorizations for hierarchical decomposition. However, these approaches impose specific sparsity patterns that may limit expressivity, and butterfly structures may suffer from noncontiguous memory access patterns that underutilize hardware resources.

In this paper, 1 we propose Low-rank Compositional Orthogonal Fine-tuning (LoCO), a novel approach that constructs orthogonal transformations through low-rank skewsymmetric matrices. Our key observation is that the tangent space to the special orthogonal group, which is the space of skew-symmetric matrices, admits a low-rank parameterization. By exploiting this structure with the ShermanMorrison-Woodbury identity [Max, 1950], we reduce the computational complexity of orthogonal fine-tuning, allowing adaptation in high-dimensional spaces. Additionally, we leverage chains of multiple low-rank rotations to enhance the expressiveness of orthogonal transformations. Rather than computing these rotations sequentially, we employ a firstorder approximation, enabling parallel computation across all rotation components. Our analysis confirms that this approximation introduces negligible error given the small-magnitude adaptations characteristic of fine-tuning. A practical advantage of LoCO is a test-time temperature scaling mechanism for controllable adaptation strength. By introducing a scalar temperature parameter, we can smoothly interpolate between pretrained and fully-adapted behaviors at inference time.

1 The extended version, including the full appendix, is available on arXiv: https://arxiv.org/abs/2605.15916

We test LoCO across three domains: (1) fine-tuning language models, (2) adapting diffusion transformers for conditional generation, and (3) fine-tuning vision transformers. Across all settings, LoCO achieves competitive or superior performance compared to established PEFT methods including LoRA [Hu et al., 2022], OFT [Qiu et al., 2023], BOFT [Liu et al., 2024b], and HRA [Yuan et al., 2024], while maintaining computational efficiency and a low memory footprint.

Contributions. Our main contributions are threefold:

• We propose LoCO, a parameter-efficient fine-tuning method that constructs orthogonal transformations via low-rank skew-symmetric matrices. We further introduce an approximation for compositional rotation chains that enables parallel computation while preserving orthogonality for fine-tuning.
• We demonstrate a test-time temperature scaling mechanism that enables flexible control over adaptation strength without retraining, providing an alternative to costly regularization schemes used in prior orthogonal methods.
• We achieve state-of-the-art or competitive results across diverse benchmarks, including GLUE, mathematical reasoning, and controllable generation.

2 RelatedWork

Low-rank Adaptation. Building on the low intrinsic rank hypothesis [Aghajanyan et al., 2020], LoRA [Hu et al., 2022] parameterizes weight updates as a product of two low-rank matrices. This avoids the inference overhead of adapters [Houlsby et al., 2019] and the optimization challenges in prompt tuning [Li and Liang, 2021; Lester et al., 2021]. Subsequent works extend this framework through adaptive rank allocation [Zhang et al., 2023; Kalajdzievski, 2023; Jiang et al., 2024], alternative initializations [Meng et al., 2024; Wang et al., 2024], and parameter sharing [Bałazy et al., 2024; Dong et al., 2023]. DoRA [Liu et al., 2024a] further decomposes updates into magnitude and direction components. Unlike this line of research, our approach applies multiplicative transformations, constructing rotation matrices from low-rank skew-symmetric parameterizations.

Orthogonal Fine-Tuning. OFT [Qiu et al., 2023; Qiu et al., 2025] parameterizes orthogonal matrices through blockdiagonal, butterfly structures [Liu et al., 2024b], structured matrices [Gorbunov et al., 2024], exploiting sparsity for efficiency. While qGOFT [Ma et al., 2024] decomposes rotations into Givens rotations, its sequential composition limits parallelization. Householder-based methods such as [Dong et al., 2024; Yuan et al., 2024; Arcas et al., 2025] use reflection parameterizations. Recent work [Liao and Monz, 2024] applies 2D rotations directly to representation subspaces. Our method differs by constructing rotations from low-rank skew-symmetric matrices, enabling efficient computation through first-order approximation while operating on higher-dimensional subspaces.

Spectral-based Methods. In another line of research, PiSSA [Meng et al., 2024] and MiLoRA [Wang et al., 2024] decompose weights via singular value decomposition. These approaches adapt residual or minor singular components selectively. Compared to these spectral-based methods, our approach preserves angular structure through orthogonal transformations rather than modifying spectral components.

Geometric Approaches. Several works exploit geometric structures: hyperbolic networks [Ganea et al., 2018; Chen et al., 2022] utilize exponential maps for hierarchical representations, and Lie group methods [Mironenco and Forr´e, 2024; Finzi et al., 2020] construct equivariant architectures. Skew orthogonal convolutions [Singla and Feizi, 2021] use matrix exponentials for training stability. While building from similar mathematical tools, these works focus on designing new architectures rather than proposing new PEFT methods.

3 Low-rank Compositional Orthogonal Fine-tuning (LoCO)

This section first reviews rotation groups and their connection to skew-symmetric matrices, and then presents the LoCO adaptation method.

3.1 Rotation Groups and Skew-symmetric Matrices

The special orthogonal group. A rotation in Rd is defined as an element of the special orthogonal group:

Here, the condition R⊤R = I indicates that rotations preserve distance, while det(R) = 1 distinguishes proper rotations from reflections. 2

Skew-symmetric matrices as tangent space. The tangent space to SO(d) at the identity is given by:

These matrices represent infinitesimal rotations or rotation velocities. The dimension of so(d) is d(d-1) 2 , corresponding to the number of independent rotation axes in d dimensions.

Matrix exponential map. The exponential map provides a canonical way to generate rotations from skew-symmetric matrices. Every rotation can be obtained from some skewsymmetric matrix. That is, given A ∈ so(d), we have

2 Reflection transformations have det(R) = -1.

This provides a smooth parameterization of rotation manifold. However, computing the matrix exponential directly is computationally expensive and would be prohibitive for the task of fine-tuning foundation models.

Cayley transform. The Cayley transformoffers an alternative approach which is more tractable for generating rotations from skew-symmetric matrices. Given A ∈ so(d), we have:

The Cayley transform is particularly well-suited in finetuning settings. During fine-tuning, we expect model adaptations to be relatively small, corresponding to rotations with limited angular magnitude. The Cayley transform provides an accurate and smooth parameterization of the rotation manifold near the identity. This approach has been successfully adopted in recent orthogonal fine-tuning methods [Qiu et al., 2023; Liu et al., 2024b].

3.2 LoCOAdaptations

This section presents the main specification of Low-rank Compositional Orthogonal Fine-tuning (LoCO).

Given a pretrained model with weight matrix W0 ∈ Rk×d , the standard linear transformation can be adapted through orthogonal fine-tuning as follows.

where R ∈ SO(d) is a special orthogonal matrix that performs rotation transformations on the input features x ∈ Rd . To enhance model expressivity, we utilize a compositional representation of R:R = R1R2 · · · Rn,

where each component Ri ∈ SO(d) represents an elementary rotation. The composition of orthogonal matrices preserves orthogonality and ensures that R ∈ SO(d). Orthogonal finetuning takes inspiration from the idea that angular features capture the semantic gap well [Chen et al., 2020; Liu et al., 2017; Liu et al., 2018; Qiu et al., 2023].

In the following subsections, we detail how to parameterize each component Ri using low-rank skew-symmetric matrices and describe the approximation scheme for the compositional rotation R. Figure 1 provides an overview of the LoCO framework.

Skew-symmetric construction

We parameterize skew-symmetric matrices through a lowrank outer product form. Given learnable matrices U,V ∈ Rd×r where r ≪ d is the rank dimension, we construct

This formula guarantees skew-symmetry since we can verify that A⊤ = (UV⊤)⊤ - (VU⊤)⊤ = VU⊤ - UV⊤ = -A. For notation convenience, we rewrite this construction using auxiliary matrices:

Efficient computation The naive computation of the Cayley transform R = (I -A)-1 (I +A) involves inverting a d × d matrix, which takes O(d3 ) time complexity. This is prohibitively expensive for high-dimensional features.

We now show how the low-rank structure of A enables a reduction in computational cost. We apply the Woodbury matrix identity [Max, 1950]. For our setting that A = XY⊤, this identity leads us to

In this case, we add subscripts for the identity matrix I to indicate the dimension of the vector space. The inversion of a d × d matrix is now converted to the inversion of a 2r × 2r matrix (I2r -Y⊤X), which requires only O(r3 ) operations. With a further algebraic manipulation, we arrive at:

The main computational cost consists of (1) computing Y⊤X ∈ R2r×2r in O(dr2 ) time, (2) inverting the 2r×2r matrix in O(r3 ) time, and (3) matrix multiplications in O(dr2 ) time. The total complexity is O(dr2 + r3 ). Since r ≪ d, typically r ∈ {2, 4, 8, . . .}, this simplifies to O(dr2 ). In practice, we adopt an input-centric execution by multiplying the input vectors with these low-rank matrices, completely avoiding the materialization of any large intermediate matrices.

Comparisonwith other orthogonal approaches. Existing orthogonal fine-tuning methods employ alternative structural constraints to manage computational costs: block-diagonal parameterization [Qiu et al., 2023] partitions transformations into independent subspaces, while butterfly structure [Liu et al., 2024b] exploits hierarchical factorizations. Our low-rank approach instead enables explicit control over the rotation subspace dimension through the rank parameter r.

Compositional Chain-of-Rotations

From a theoretical perspective, any rotation in Rd can be decomposed into a sequence of Givens rotations in 2dimensional planes. Specifically, any element of SO(d) can be written as a product of at most d(d-1) 2 Givens rotations [Press, 2007]. Our chain-of-rotations framework can be viewed as a learnable, parameter-efficient approximation to this decomposition, where each component rotation Ri operates on a distinct low-dimensional subspace determined by Ui, Vi .

We define a chain of n rotations:

where each Ri is parameterized by its own pair Ui ∈ Rd×r ,Vi ∈ Rd×r . Since the composition of orthogonal matrices remains orthogonal, R ∈ SO(d) is guaranteed.

To enable efficient parallel computation, we use a firstorder approximation by expanding the product and retaining only linear terms:

This approximation replaces sequential matrix multiplication with a sum of low-rank updates, which can be computed in parallel using standard GPU operations.

Approximation error analysis. Consider rotations where ∥Xi∥F , ∥Yi∥F = O(ε) for small ε > 0. Each perturbation ∆i = 2Xi(I - Y⊤ i Xi)-1Y⊤ i has magnitude O(ε2 ), since it involves the product of Xi and Y⊤ i . When expanding the product:

Here, the second-order cross terms ∆i∆j are O(ε4 ) and highorder terms are at least O(ε6 ). During fine-tuning, where parameters are initialized near zero and remain small throughout training, these higher-order terms contribute negligibly to the transformation.

Orthogonality. We numerically evaluate the first-order approximation by measuring how well it preserves vector norms. For randomly initialized vectors x, we compute the relative error |∥x∥-∥Rx∥|/∥x∥ across ∥Xi∥F = ∥Yi∥F = ε for ε ∈ [10-6 , 10-0.5 ], a range typical of fine-tuning perturbations. Figure 2 shows that our approximation effectively preserves vector magnitude (ideally ∥Rx∥ = ∥x∥) across the range of ε values.

Remark. The first-order approximation enables all n rotations to be computed independently and summed, fully utilizing GPU parallelism, while simplifying backpropagation compared to sequential matrix products. While compositional orthogonal transformations appear in BOFT [Liu et al., 2024b] (butterfly factorization) and reflection-based methods [Yuan et al., 2024], neither leverages parallel computation. BOFT uses sequential butterfly stages, and reflection chains require multiple compositions. Our outer-product parameterization with first-order approximation uniquely enables fully parallel execution. This is also reflected by our experimental validation in Section 4.

4 Experimental Results

We evaluate LoCO on three tasks: fine-tuning large language models, diffusion transformers, and vision transformers.

4.1 Large Language Model Adaptation

We examine LoCO on two tasks: natural language understanding (NLU) and mathematical reasoning.

Baselines. We compare against orthogonal fine-tuning methods: (a) OFT [Qiu et al., 2023], which uses blockdiagonal orthogonal matrices; (b) BOFT [Liu et al., 2024b], which employs butterfly factorizations to construct dense orthogonal adapters; and (c) HRA [Yuan et al., 2024], which chains Householder reflections [Householder, 1958].

Natural Language Understanding

Following [Qiu et al., 2023; Liu et al., 2024b; Yuan et al., 2024], we fine-tune DeBERTa-V3-base [He et al., 2021] on GLUE benchmark tasks.

Setup. We adopt the experimental setup of BOFT and HRA, applying each method to all linear modules within transformer layers. We increase OFT's block size from b=16 to b=24 to match parameter counts, and use HRA's best configuration (λ=0). Additional details are in Appendix.

Results. Table 1 reports matched accuracy for MNLI, Matthews correlation for CoLA, Pearson correlation for STSB, and accuracy for other tasks. LoCO achieves the best average score (87.75), outperforming HRA (85.64) while using 30% and 10% fewer parameters than OFT and BOFT, respectively. LoCO ranks first on 5 out of 8 tasks (SST-2, MRPC, QNLI, RTE, STS-B). Notably, HRA shows a substantial drop on CoLA, suggesting Householder reflections may be less stable across diverse linguistic tasks than orthogonal rotations.

Mathematical Reasoning

Setup. We fine-tune LLaMA2-7B [Touvron et al., 2023] on MetaMathQA-40K [Yu et al., 2023], adapting only the query and value projections (Wq, Wv). All baselines are reproduced under identical settings [Yuan et al., 2024; Liu et al., 2024b] and evaluated on the GSM8K [Cobbe et al., 2021] and MATH [Hendrycks et al., 2021] validation sets.

Results. As shown in Table 2, LoCO achieves the highest GSM8K score (50.19), outperforming OFT (49.35), HRA (49.65), and BOFT (45.58) with only 0.12% trainable parameters. On MATH, LoCO matches OFT while surpassing BOFT and HRA. These results suggest that low-rank skewsymmetric parameterization captures mathematical reasoning patterns more effectively than block-diagonal (OFT/BOFT) or Householder (HRA) structures.

Computational Efficiency

We compare training efficiency by measuring time per step and peak memory after 100 warm-up steps, averaged over 300 iterations across 3 trials. Experiments are conducted on DeBERTa-V3 (184M) and LLaMA2-7B with matched parameter counts. Details are provided in Appendix.

As shown in Figure 3, LoCO matches the efficiency of sparse OFT while outperforming dense methods (HRA, BOFT). Under heavy configurations, HRA incurs 160% longer step time and 20% higher memory usage, whereas LoCO maintains stable throughput. This efficiency stems from two factors: (1) the Sherman-Morrison-Woodbury formula reduces matrix inversion to O(r3 ) complexity, and (2) all rotation components compute in parallel, unlike HRA's sequential backward pass or BOFT's non-contiguous memory access patterns [Dao et al., 2022].

Summary. Across NLU and mathematical reasoning benchmarks, LoCO achieves competitive or superior performance while maintaining both parameter efficiency and computational scalability.

4.2 Fine-tuning Diffusion Transformers

Setup. We follow the setup of [Tan et al., 2025] focusing on conditional generation tasks. Similar to the OminiControl setup, we use FLUX.1 [Labs, 2024] as the pre-trained model. We conduct all experiments on 4 NVIDIA H200 GPUs. To match the total samples of OminiControl [Tan et al., 2025], we train our models for 25,000 iterations (half of the setting of [Tan et al., 2025] when running on 2 GPUs).

Baselines. We compare our approach against two representative baseline methods: LoRA [Hu et al., 2022]-the default adapter of OminiControl [Tan et al., 2025], serving as a non-orthogonal adapter-and butterfly orthogonal finetuning (BOFT) [Liu et al., 2024b] as an orthogonal adapter. All baseline experiments utilize the official OminiControl implementation. We extend this framework to incorporate BOFT [Liu et al., 2024b] into the FLUX pipeline. Our LoCO adapter is implemented within the HuggingFace PEFT library to ensure seamless integration with the existing codebase.

We evaluate performance using established metrics: Fr´echet Inception Distance (FID) [Heusel et al., 2017], Structural Similarity Index (SSIM), CLIP-IQA [Wang et al., 2023], MANIQA [Yang et al., 2022], MUSIQ [Ke et al., 2021], and Peak Signal-to-Noise Ratio (PSNR) [contributors, 2025]. For controllability assessment, we adopt the OminiControl approach, measuring the F1 score between extracted and input edge maps for edge-conditioned generation, and Mean Squared Error (MSE) between extracted and original condition maps for other tasks. To evaluate consistency between prompts and generated images, we use CLIP Text-Image similarity. Following OminiControl [Tan et al., 2025], we evaluate on 5,000 images from the COCO dataset, with prompts extracted from corresponding image captions.

Results. Table 3 presents quantitative results across spatially-aligned controllable generation tasks. Our method demonstrates competitive performance across the evaluation landscape. Specifically, on image quality metrics (CLIP-IQA, MAN-IQA, MUSIQ), our approach consistently achieves performance comparable to or exceeding the baselines. More significantly, our method exhibits superior alignment fidelity as measured by CLIP Image scores across the majority of tasks. For instance, on the Mask task, our method achieves a CLIP Image score of 0.824, outperforming BOFT (0.801) and LoRA (0.812). Similarly, on the Deblur task, we obtain 0.832 compared to 0.803 for BOFT. These results indicate that our low-rank compositional rotation approach effectively preserves semantic alignment. Qualitative examples of these methods are provided in Appendix.

Test-time adjustable generation. We introduce a temperature parameter t for the skew-symmetric matrix A, whereby the rotation matrix is computed as R = (I -At)-1 (I +At). During training, we use t = 1. At inference time, t can be adjusted to control the degree of adaptation. Since the scaled matrix At remains within the tangent space so(d) for varying values of t, the resulting transformations continue to yield semantically meaningful outputs in the generated images. This property is preserved because the Cayley transformation provides a good approximation to exp(At) given a small magnitude of A.

This adjustable inference mechanism described here has not been explored by prior orthogonal fine-tuning approaches [Qiu et al., 2023; Liu et al., 2024b]. Furthermore, orthogonal fine-tuning methods based on reflection transformations, such as Householder-based approaches [Yuan et al., 2024], do not possess this capability due to their different geometric structure.

Figure 4 illustrates the visual quality of generated images across a range of temperature values t ∈ [0, 2]. Additional figures demonstrating generated images beyond this range using LoCO are provided in the Appendix. Compared with images generated by varying the scaling parameter α in LoRA for OminiControl (the two bottom rows in Figure 4), our method exhibits robustness and stability across different scaling factors. This is verified quantitatively by image quality metrics in Figure 5. The generation quality when varying t is more robust and less brittle than LoRA.

Effect of the number of rotations n. We ablate the number of rotations (using values of 2, 4, 6, 8) on the Canny edge detection setting. Table 4 shows the results. We observe that the model benefits from a higher number of rotations, which increases the expressiveness of the LoCO adapter. However, since each rotation approximation can be computed in parallel, GPU memory increases linearly with the number of rotations.

4.3 Fine-tuning Vision Transformers (ViTs)

Dataset. We conduct experiments fine-tuning vision models on the Visual Task Adaptation Benchmark (VTAB). Specifically, we employ VTAB-1k [Zhai et al., 2019], which comprises 19 diverse visual classification tasks organized into three distinct groups: Natural (tasks featuring natural images), Specialized (medical and satellite imagery), and Structured (tasks requiring geometric understanding). In the finetuning setting, the number of training samples for each task is constrained to 1,000 examples. The test dataset for evaluation remains consistent with the original VTAB test set.

Pretrained model. We use ViT-B/16 [Dosovitskiy, 2020] as our main backbone, which has been pretrained on the large-scale ImageNet-21K dataset.

Baselines. We compare our method against two distinct categories of baselines: non-orthogonal transformation methods and orthogonal adaptation methods. For nonorthogonal baselines, we include LoRA [Hu et al., 2022], which decomposes weight updates using low-rank matrices; Adapter [Houlsby et al., 2019], which inserts bottleneck modules; Scale-and-Shift Feature (SSF) [Lian et al., 2022], which performs lightweight feature transformations; and Visual Prompt Tuning (VPT) [Jia et al., 2022], which learns task-specific prompt embeddings. For orthogonal finetuning baselines, we compare with BOFT [Liu et al., 2024b] and HRA [Yuan et al., 2024]. We perform a hyperparameter search for the learning rate of all models from the set [5e-5, 1e-4, 5e-4, 1e-3] and report the best test performance for each method.

Results. Figure 6 shows the performance across models. LoCO achieves competitive accuracy across all three VTAB1k benchmark categories, with the highest average among orthogonal PEFT methods on Structured tasks (63.4%) while remaining competitive on Specialized tasks (86.4%) and Natural tasks (82.3%). On Natural images, LoCO obtains strong results on Caltech101 (95.6%), DTD (71.4%), and Flowers102 (99.4%). Notably, LoCO shows substantial gains on out-of-distribution tasks: PatchCamelyon reaches 88.1% versus BOFT's 83.2% (+4.9%), demonstrating effective transfer to medical imaging. Performance on EuroSAT (95.6%) and RESISC45 (85.4%) further confirms robustness on specialized domains. Detailed per-dataset results are provided in Appendix.

5 Conclusion

We introduce a low-rank compositional orthogonal finetuning (LoCO), a parameter-efficient method that adapts foundation models through orthogonal transformations. By constructing orthogonal matrices via low-rank skewsymmetric parameterization and leveraging the Cayley transform with the Sherman-Morrison-Woodbury identity, LoCO reduces computational complexity while allowing fully parallel computation of compositional rotations. Experiments across natural language understanding, mathematical reasoning, diffusion transformers, and vision transformers demonstrate consistent improvements over existing orthogonal finetuning methods.

Limitations and future work. While LoCO's low-rank formulation addresses scalability for typical fine-tuning scenarios, high-dimensional settings may benefit from further optimization. Future directions include theoretical analysis of optimal rotation decompositions, extending temperature scaling to advanced applications such as combining multiple adaptations with multiple scaling factors, and application to multimodal models.

Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS2019-II190079, Artificial Intelligence Graduate School Program (Korea University);RS-2019-II190075, Artificial Intelligence Graduate School Program (KAIST); No. RS-2022II220984, Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation; No. RS-2024-00457882, AI Research Hub Project), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS2025-23525509) and the High-Performance Computing Support Project, funded by the Government of the Republic of Korea (Ministry of Science and ICT) (No. RQT-25-070263).