Abstract
Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method that facilitates the lightweight adaptation of large language models (LLMs) by introducing low-rank update matrices, and has since motivated the development of various extensions. However, LoRA and its most variants overlook the issue of information loss in deep LoRA layers, where signals or gradients passing through multiple LoRA blocks may gradually vanish before reaching the final layers of the network. This limitation hampers convergence speed during fine-tuning. To address these challenges, we propose KnitLoRA, an innovative dense connection low-rank adaptation framework. First, we introduce dense connections between each LoRA block and multiple, or even all preceding blocks to facilitate feature reuse and fusion. Second, these connections improve gradient flow and mitigate the vanishing gradient problem. Third, KnitLoRA accelerates model convergence, further enhancing its efficiency in fine-tuning. Importantly, KnitLoRA eliminates dense connection paths during inference, incurring no extra computational overhead. Our experiments show that KnitLoRA outperforms LoRA and its variants without adding any additional trainable parameters, achieving stronger performance and faster, lower loss reduction. Codes and models will be available online.
Keywords: Large language model, Parameter-efficient fine-tuning, Low-rank adaptation, Multimodal learning
Subject terms: Engineering, Mathematics and computing
Introduction
Large Language Models (LLMs) have demonstrated remarkable performance, but also come with substantial training costs. LLMs focus on broad general knowledge during initial training, but may lack sufficient accuracy in specific domains or tasks. Therefore, they are often fine-tuned for downstream tasks. However, Full Fine-Tuning (FFT) requires enormous resource consumption, leading to the development of Parameter-Efficient Fine-Tuning (PEFT)1,2, which reduces computational overhead while improving fine-tuning efficiency. Low-Rank Adaptation (LoRA)3, as a form of PEFT, aims to minimize the parameters that need to be adjusted, achieving efficient fine-tuning of LLMs by adding two low-rank matrices next to the frozen linear layers of the original model and fine-tuning these two matrices. Building on LoRA, researchers have proposed several variants4,5 to optimize the performance-resource trade-off across different task, expanding the scope of the application of LoRA.
Although LoRA has demonstrated excellent performance in domain adaptation, the gradient must pass sequentially through each LoRA block, which can lead to exponential decay and vanishing gradients in very deep networks. There are two types of LoRA variants. The first category, parameter-efficient LoRA methods (e.g., LoKr6, AdaLoRA5), focus on parameter reduction but neglect gradient propagation issues. The second, structure-efficient LoRA methods (e.g., ResLoRA4), address this via structural adjustments like residual connections between LoRA blocks. However, all of these approaches still face limitations. First, overly sparse connections result in insufficient multi-level feature fusion for deeper domain adaptation. This constraint limits the model’s capacity to capture hierarchical representations essential for effective fine-tuning on complex downstream tasks. Second, connecting multiple entire LoRA blocks increases training overhead and overlooks gradient propagation issues within blocks. This inefficiency can hinder stable optimization during large-scale model fine-tuning, especially when adapting to diverse downstream domains. The lack of multi-level information exchange between LoRA blocks could affect the domain knowledge transferring. Consequently, this localized approach requires more training iterations to capture sufficient global features, and we believe that there is still considerable room for improvement in the convergence speed of many LoRA structures.
To address those issues, we propose KnitLoRA, a dense connectivity–enhanced low-rank adaptation framework that restructures the network for lightweight fine-tuning of LLMs. This methodology establishes dense connections between each LoRA block and multiple preceding blocks for deeper fine-tuning of reasoning. Considering the influence of different connection patterns, we develop three distinct KnitLoRA structures. For the no-plain structure (where additional connections modify the forward computation path), we implement parameter merging strategies that enable seamless conversion to standard LoRA during inference, eliminating computational overhead from auxiliary pathways. Additionally, we construct the HSCoT benchmark by extending HellaSwag8 with Chain-of-Thought annotations generated via DeepSeek-R1-671B9, specifically designed to assess logical reasoning capabilities. Comparative experiments against LoRA and its variants demonstrate that KnitLoRA achieves superior inference performance and accelerated loss convergence. Furthermore, our text-to-image experimental results demonstrate KnitLoRA’s exceptional advantages in image generation, further highlighting the multimodal learning capabilities of our architecture beyond traditional language processing. Figure 1 depicts the architectural framework of KnitLoRA alongside our experimental tasks and findings, collectively demonstrating the efficacy of our proposed approach. Finally, the parameter merging methods ensures compatibility with existing LoRA variants, facilitating adoption across diverse LoRA frameworks.
Fig. 1.
Structures of KnitLoRA and tasks used in our experiment, along with its performance on SVAMP7. It can be observed that KnitLoRA achieves higher accuracy and faster convergence speed compared to LoRA3 and ResLoRA4.
In summary, the contributions of our work are as follows:
We propose KnitLoRA, an innovative low-rank adaptation framework that introduces dense connections between LoRA blocks. KnitLoRA not only accelerates model convergence but also significantly enhances model’s semantic reasoning performance. We also analyze the reasons for these improvements.
We design three KnitLoRA structures, each of which has advantages for different models and tasks. Furthermore, we use merging approaches to convert them into standard LoRA structures during inference, maintaining the efficiency of the inference process, making this framework easily applicable to other LoRA methods.
We conduct experiments on multiple tasks and models, comparing our approach with LoRA and its variants, demonstrating effectiveness of KnitLoRA while showing its advantages in the multimodal learning. Confirming that KnitLoRA enables deeper semantic reasoning.
We introduce the HSCoT dataset in the domain of commonsense reasoning, comprising 1.5K samples and integrating Chain-of-Thought (CoT) into the HellaSwag dataset to evaluate models’ logical reasoning capabilities.
Related works
Parameter-efficient fine-tuning (PEFT)
Certain PEFT strategies effectively minimize trainable parameters through the integration of trainable modules or compact layers1. Others selectively optimize specific model parameters while maintaining others in a frozen state10. For example, the FineDiffusion11 method performs fine-grained image generation by fine-tuning only the extended embedding layers, bias terms, and normalization parameters, achieving competitive or superior performance compared to full-parameter fine-tuning while training only 1.77% of the parameters and attaining a 56% speedup. Approaches such as LoRA3, utilize low-rank reparameterization to decrease adaptation complexity, while Prefix Tuning2 achieves adaptation to downstream tasks by introducing trainable prefix vectors. Despite their demonstrated benefits, these approaches face inherent limitations in adaptation efficiency and architectural complexity, motivating subsequent structural innovations to improve optimization effectiveness.
LoRA’s variants
The original LoRA framework3 exhibits performance constraints arising from low rank configurations, frequently necessitating extended training iterations to attain optimal results. Recent studies have investigated adaptive rank determination strategies, with LoKr6 proposing Kronecker-based decomposition for more flexible parameter allocation. Alternative approaches emphasize hybrid architectures: HydraLoRA12 combines Mixture of Experts with LoRA for multi-task learning efficiency, while ResLoRA4 incorporates residual connectivity mechanisms into LoRA modules to accelerate convergence dynamics. Despite these advancements in parameter optimization and architectural integration, existing LoRA variants systematically neglect critical challenges in deep adaptation stacks–particularly the degradation of cross-block information flow and limited feature reutilization across sequential LoRA layers. To address these limitations, we propose integrating dense connectivity patterns from DenseNet13 into the LoRA framework, enabling comprehensive feature interaction while preserving parameter efficiency.
Densely connected neural network
Research in related domains has explored similar challenges, particularly in image feature extraction. Notably, DenseNet13, as a Convolutional Neural Network (CNN), improves gradient propagation and information flow through dense inter-layer connectivity, alleviating vanishing gradients and enabling efficient feature reuse. It achieves higher accuracy than ResNet with fewer parameters. Owing to its compact and effective design, DenseNet has been extended to diverse applications: MDU-Net14 enhances biomedical segmentation, DCCT15 integrates dense connections into Transformer architectures for image dehazing, and16 introduces a tensor network-based DenseNet with a tensor regression layer to reduce memory consumption while maintaining high classification accuracy. Furthermore, DenseAttNet17 incorporates a self-attention mechanism into a 3D DenseNet for multi-label emotion recognition, demonstrating its potential for high-dimensional spatiotemporal modeling. However, these methods require fully training dense feature extraction architectures in both the pre-training and fine-tuning stages, exhibiting limited flexibility for structural adjustment during LLM fine-tuning. In contrast, KnitLoRA restructures the network to enable lightweight adaptation of LLMs across diverse downstream tasks.
Methodology
This section details implementation of KnitLoRA framework, which comprises two phases: During the training phase (section 3.2), we construct KnitLoRA blocks through three distinct dense connectivity structures, each designed to optimize specific adaptation characteristics. In the inference phase (section 3.3), we implement parameter merging techniques to eliminate dense connectivity paths in the no-plain structure variant, thereby preserving computational efficiency while maintaining performance gains achieved during training.
Preliminary
The core mechanism of LoRA involves inserting low-rank parameter matrices within designated network layers to constrain gradient updates during fine-tuning. This approach substantially reduces computational and storage overhead by limiting trainable parameters to auxiliary matrices. Consider a linear layer in a pre-trained model with weight matrix 
, where 
and 
represent output and input dimensions. Conventional fine-tuning modifies the entire weight matrix W, while LoRA introduces parameter-efficient adaptation through decomposition into two low-rank matrices:
 |
1 |
and 
constitute low-rank adaptation matrices with rank 
. This formulation drastically reduces trainable parameters while preserving model capacity. The forward propagation equation becomes:
 |
2 |
where x denotes input tensor, h represents output tensor, and n indicates the layer index within the network architecture.
In ResLoRA, the input of each LoRA block is derived from the current layer’s input tensor 
and incorporates the preceding layer’s input tensor 
. This architectural design is inspired by ResNet18, where residual connections facilitate gradient propagation through direct pathways between consecutive layers. The mathematical formulation of the ResLoRA mechanism can be explicitly represented as follows:
 |
3 |
KnitLoRA structures
Building on DenseNet’s success in enhancing gradient propagation and feature reutilization through dense connectivity, we propose three KnitLoRA architectures that establish multi-path information flow between LoRA blocks for deeper fine-tuning of semantic reasoning. These structures are: input-based dense connectivity (In-Dense), A-matrix-based connectivity (
), and B-matrix-based connectivity (
), corresponding to 
, 
, and 
. Each structural variant implements distinct feature aggregation mechanisms while maintaining parameter efficiency. The architectural details are visualized in Figure 2, highlighting the dense connections of different KnitLoRA implementations.
Fig. 2.
Structures of KnitLoRA. We present three structures introduced in section 3.2. We introduce dense connections into the input, A-matrix, and B-matrix. And we refer to them as (a) KnitLoRA
, (b) KnitLoRA
, and (c) KnitLoRA
, respectively.
Input-based dense connection structure implements dense connectivity by integrating input signals from all preceding LoRA blocks into the current block’s input processing. Distinct from DenseNet’s concatenation/summation operations, KnitLoRA avoids direct output tensor manipulation due to dimensional incompatibility between LoRA blocks. Constructing dense connections at output tensor level would necessitate additional dimension alignment layers, introducing superfluous parameters that undermine fine-tuning efficiency. Furthermore, the original model contains multiple functional layers between consecutive LoRA blocks, bypassing these components through direct output reuse would disrupt hierarchical feature extraction. To address these, KnitLoRA establishes feature fusion pathways within the input domain while preserving architectural integrity. The computational formulation is:
 |
4 |
Figure 2aillustrates the architectural implementation of this input-domain dense connectivity pattern.
A-matrix-based dense connection structure implements dense connectivity through direct aggregation of A-matrix parameters across sequential LoRA blocks. In standard LoRA, the A-matrix operates closer to input tensors compared to B-matrices, while native Transformer architectures incorporate inherent shortcut pathways such as residual connections. This suggests inherent compatibility between A-matrix adaptation and existing shortcut pathways, a principle previously demonstrated in ResLoRA. Building on this observation, our A-Dense structure enhances B-matrix adaptability by integrating current and preceding A-matrices from m historical LoRA blocks (
) into the computation process. The aggregated A-matrix outputs are summed before interacting with the B-matrix, providing indirect shortcut pathways for B-matrix adaptation while minimizing structural modifications to A-matrix operations. The formulation becomes:
 |
5 |
This architectural design, visualized in Figure 2b, demonstrates performance sensitivity to 
values where connection density directly impacts feature representation quality. Detailed ablation analysis regarding optimal connection depth is provided in section4.3.
B-matrix-based dense connection structure implements dense connectivity through parameter-level aggregation of B-matrices across sequential LoRA blocks. While A-matrix aggregation demonstrated performance improvements, the B-matrix as the terminal component in low-rank projection, governs feature reconstruction quality. Given the limitations of single low-rank layers (
) in capturing complex transformation patterns, we enhance representational capacity by integrating B-matrix parameters from m preceding LoRA blocks (
). Through parameter-level fusion of current and historical B-matrices, this approach enables cross-layer feature complementarity while preserving output dimensionality consistency. The formulation becomes:
 |
6 |
Figure 2cillustrates this parameter aggregation architecture, where hyperparameter 
governs the depth of historical block integration. This design preserves original LoRA’s structural constraints while enhancing projection expressiveness through multi-path parameter aggregation.
KnitLoRA introduces three dense connectivity patterns across LoRA blocks, improving feature reutilization and parameter capacity while maintaining constant parameter counts. The established dense pathways facilitate accelerated gradient propagation to deeper network segments during backpropagation, thereby demonstrating the framework’s structural benefits through enhanced optimization dynamics and reduced signal attenuation in multi-layer configurations.
Parameter merging
Now, let us turn our attention to the A-Dense and B-Dense architectures. It is observed that, because of the nonexistence of extra forward propagation routes in these two structures, the direct integration of LoRA block weights is made possible. After the merging of 
and 
, the weights 
can be represented by Equations 7 and 8:
 |
7 |
 |
8 |
The mergeable characteristics of A-Dense and B-Dense structures guarantee zero inference overhead weight consolidation. However, the In-Dense structure introduces supplementary forward pathways. Specifically, the summation of input tensors from preceding LoRA blocks (
), which prevents weight merging. This architecture corresponds to the “non-plain” structure in ResLoRA, whereas A-Dense and B-Dense belong to “plain” structures. The additional forward pathways in In-Dense operate independently from the base model during inference, introducing computational overhead that undermines LoRA’s parameter efficiency. While ResLoRA proposed two merging strategies for similar challenges, our dense connectivity patterns exhibit greater complexity due to non-ResNet-like architectural dependencies. We implement customized adaptations of these merging strategies to better align with our framework’s requirements, ensuring elimination of dense connections during inference phase.
For the non-plain structure, a scaling factor 
must be determined such that 
, enabling transformation of dense pathways into original computation paths. However, varying 
and 
yield distinct 
values, necessitating approximation of an optimal 
to model their relationship. Applying this principle to the In-Dense structure, the forward propagation formulation (Equation 9) and parameter merging equation (Equation 10) can be derived:
 |
9 |
 |
10 |
For 
estimation, we adopt the mainstream methodologies categorized into input-driven merging (merge
) and parameter-driven merging (merge
)4, with structural adaptations for compatibility. Prior studies have validated the Frobenius norm as an metric for quantifying matrix magnitude19. Both KnitLoRA merging strategies estimate 
using this norm-based analytical framework to eliminate dense connectivity paths during inference phase.
Dense-Input-Merge refers to a technique for computing 
on the basis of the input tensors 
and 
. In this method, the quantity of sliding windows in merge
is raised to two. These windows serve to take in the most recent input tensor 
and 
. Then, we work out the average Frobenius norm of every tensor inside each sliding window. This norm is employed to gauge the magnitude of 
and 
. Suppose that 
stands for the average Frobenius norm of the input tensor 
, and 
represents the average Frobenius norm of the 
tensor. Then, the subsequent mathematical relation is valid:
 |
11 |
Consequently, the formula for 
is deduced as follows:
 |
12 |
Dense-Parameter-Merge is a technique developed from merge
that computes 
according to the weights of previous LoRA blocks. As stated in merge
, 
can be utilized to represent the magnitude of 
. On this basis, by disregarding the impact of the input tensor and concentrating on the effect of the weights, the Frobenius norm of the merged weight from the previous block can serve as a representation of the magnitude of 
. We signify the Frobenius norm of the merged weight at the n-th block as 
. The mathematical relationship can be presented as:
 |
13 |
Owing to the dense connection of KnitLoRA, we are unable to directly calculate 
. 
has to be divided and computed for each dense connection path. The formula of 
is:
 |
14 |
Through the proposed merging strategies, we systematically resolve computational overhead introduced by dense connectivity pathways during inference. The two parameter consolidation approaches are designated as DI-Merge (Dense-Input-Merge) and DP-Merge (Dense-Parameter-Merge), ensuring structural compatibility with standard LoRA deployment while eliminating redundant forward propagation paths through optimized weight transformation mechanisms.
Why KnitLoRA accelerates training
Although from the perspective of feature utilization and gradient propagation, the KnitLoRA structure enables effective feature reuse and allows gradients to reach deeper layers more quickly during backpropagation, addressing the vanishing gradient problem in deep networks, which intuitively explains why our structure accelerates model training. This intuition can be further supported by the following mathematical analysis, for the LoRA block, the gradient calculation sub-equation is:
 |
15 |
However, for the In-Dense structure, taking the third LoRA block as an example, where the number of preceding LoRA blocks is 2, the equation for 
is:
 |
16 |
By comparing Eq. (15) with Eq. (16), it can be observed that the last term in the sub-equation of the In-Dense structure has no additional factors. This is because the cross-layer connection bypasses intermediate layers, preventing potential gradient vanishing or explosion issues in the factors 
and 
, thus accelerating the model convergence process.
HSCoT benchmark construction
In GenKP20, structured triples from a knowledge graph are generated via in‑context learning, then weighted, filtered, and injected into a large language model; experimental results demonstrate that this approach yields higher accuracy across multiple NLP tasks compared to traditional template‑based or pure‑text prompting. ReLMKG21, by contrast, utilizes a graph neural network to perform multi‑hop explicit reasoning over the knowledge graph, tightly integrating textual representations with graph structure, and likewise significantly outperforms methods relying solely on text retrieval. Together, these studies demonstrate that embedding explicit structured reasoning within large language models far surpasses the “example–fill‑in‑the‑blank” style of pure‑text prompting, and have catalyzed the development of Chain‑of‑Thought (CoT) reasoning techniques–whereby a step‑by‑step reasoning chain is articulated within the prompt to furnish the model with fine‑grained thought pathways, thereby further enhancing performance on complex reasoning tasks.
Given that KnitLoRA
demonstrated significant advantages on the HellaSwag dataset, we sought to further evaluate its reasoning capabilities by introducing HSCoT benchmark. HSCoT consists of 1.5K samples, among which 1 K training samples are taken from the HellaSwag training set and augmented with CoT annotations, as depicted in Figure 3. The CoT annotations were initially generated by DeepSeek-R1-671B9 through sample-specific prompts and subsequently validated by annotators, the prompt is as shown in Figure 4. The 500 test samples are directly retained from HellaSwag’s evaluation set. By measuring model performance on HSCoT after fine-tuning, we quantitatively assess CoT reasoning acquisition capabilities. As shown in Table 1, only KnitLoRA achieved meaningful improvements over baseline methods, with statistically significant margins confirming the architectural advantages in capturing structured reasoning patterns.
Fig. 3.
Samples from the HSCoT dataset, with correct reasoning and analysis of incorrect options within the CoT framework.
Fig. 4.
The methodological framework for CoT-driven data augmentation. We use DeepSeek R1 in the implementation.
Table 1.
Results of Fine-tuning LLaMA3.2 on HellaSwag and HSCoT. ‘/’ indicates that the model failed to converge.
| Method | Size | Task | LoRA | AdaLoRA | LoKr | ResLoRA
|
KnitLoRA
|
|---|---|---|---|---|---|---|---|
| LLaMA3.2 | 1B | HellaSwag | 5.72 | / | / | 14.28 | 35.39 |
| HSCoT | 5.4 | / | / | / | 27.8 |
Code availability
All code and data used in this study are available from the archived repository with the DOI: 10.5281/zenodo.19212918, and are released under the MIT License.
Experiments
Experimental setup
Our evaluation spans three representative domains: natural language generation (NLG), natural language understanding (NLU), and text-to-image generation (T2I), employing multiple model architectures across these tasks. Through comparative analysis against established methods including LoRA3, ResLoRA4, AdaLoRA5, LoKr6, and MTL-LoRA22, we systematically validate KnitLoRA’s performance improvements in language comprehension, generative capabilities, logical reasoning, and cross-modal adaptation. We consider only linear layers for the target modules of LoRA blocks.
Details of NLG. Following the experimental settings of LoRA3, we set the rank 
, 
, and selected the target modules as 
and 
. We also standardized the connection density of KnitLoRA
and KnitLoRA
with KnitLoRA
, seting the 
. For all datasets, we report accuracy as the evaluation metric. To prevent merging methods from affecting the true performance of our structure, we did not enable any merging method for KnitLoRA
in the experiments (ResLoRA
is treated similarly). We employed the ZeRO-2 stage via Deepspeed23 to offload unnecessary parameters from the GPU to the CPU. ResLoRA
means we use the Input-shortcut structure of ResLoRA. We employ LLaMA3.2-1B24, Qwen2-1.5B25 and LLaMA2-7B26 as base models for NLG experiments. Systematic evaluation against comparative LoRA frameworks utilizes six datasets, assessing model performance across mathematical reasoning (GSM8K27, SVAMP7, MetaMathQA28, MathQA29) and commonsense reasoning (HellaSwag8 and our proposed HSCoT). Datasets specifications are summarized in Table 2.
Table 2.
Details of datasets used in NLG experiments. Reasoning process refers to the presence of semantic reasoning steps for model training. Open-ended indicates whether the answer is provided as options or needs to be generated.
| Dataset | Examples | Domain | Reasoning Process | Open ended |
|---|---|---|---|---|
| SVAMP | 1,000 | Math | Yes | Yes |
| GSM8K | 8,500 | Math | Yes | Yes |
| MathQA | 37,200 | Math | Yes | No |
| MetaMathQA | 395,000 | Math | Yes | Yes |
| HellaSwag | 59,950 | Commonsense | No | No |
| HSCoT | 1,500 | Commonsense | Yes | No |
Details of NLU. Our NLU experiment configuration follows ResLoRA4 and fine-tuning RoBERTa-large30, while we did not enable Deepspeed. We apply DoRA to 
, 
, 
, 
and the two-layer linear transformation matrices 
, 
of the feed-forward network (FFN), and use two ranks 
and 
. The remaining hyperparameters are set the same as NLG tasks. We report the Matthew correlation coefficient for CoLA, the Spearman correlation coefficient for STS-B, and accuracy for other tasks. Our NLU evaluation methodology follows3. We implement RoBERTa-large30, an established open-source LLM, as the base architecture within the GLUE31 benchmarking framework. GLUE’s comprehensive metrics are widely adopted as standard benchmarks for evaluating language model cognition, and we select seven representative tasks from its evaluation suite to systematically assess model performance across diverse linguistic capabilities.
Details of T2I. Consistent with the ResLoRA4, we applied method to the U-Net32 module of the Stable-Diffusion model. We added LoRA blocks to all 
, 
, 
, 
layers and set the rank r to 16 and 
to 32. Each pair of input text and target images was manually verified and filtered. This experiment utilizes the established Stable-Diffusion-v2 framework33 for text-to-image generation tasks, designed to assess architectural capabilities in producing cartoon-style visual outputs. The Tuxemon dataset34, a Pokémon-inspired collection featuring textual descriptions paired with cartoon-style illustrations, serves as the evaluation benchmark. Model pre-training was conducted on real-image corpora, followed by domain-specific fine-tuning using cartoon-style image samples. Performance assessment focuses on visual fidelity metrics comparing generated outputs against reference cartoon-style characteristics.
Main results
NLG performance
Table 3 summarizes the empirical evaluations comparing KnitLoRA against baseline methods across diverse tasks and model architectures. Excluding the GSM8K task for 1B model fine-tuning, KnitLoRA
and KnitLoRA
demonstrate superior performance over all baseline approaches. For instance, when adapting LLaMA2-7B on MetaMathQA, the three KnitLoRA variants achieve performance gains of 99.3%, 85.8%, and 15.9% respectively compared to standard LoRA. As detailed in Table 1, the B-Dense structure (KnitLoRA
) attains a 35.39% absolute improvement on HellaSwag, substantiating the efficacy of dense parameter aggregation in complex reasoning scenarios. Additionally, since all architectures achieved high accuracy on SVAMP, we further validated the effectiveness of our architecture using this task on Qwen2-1.5B. As shown in Figure 5, the KnitLoRA architecture demonstrates significant advantages in both convergence speed and accuracy. These quantitative results validate the structural advantages of dense connectivity patterns in low-rank adaptation frameworks.
Table 3.
Main Results of Fine-tuning LLaMA-3.2 and LLaMA-2 on NLG tasks. MMQA stands for MetaMathQA. Optimal values are bolded.
| Model | Size | Method | GSM8K | SVAMP | MMQA | MATHQA |
|---|---|---|---|---|---|---|
| LLaMA3.2 | 1B | LoRA3 | 15.85 | 54.67 | 20.85 | 26.06 |
| AdaLoRA5 | 9.55 | 37.33 | 11.98 | 23.38 | ||
| LoKr6 | 9.78 | 45.33 | 12.13 | 21.44 | ||
ResLoRA 4
|
17.89 | 55.67 | 21.38 | 24.82 | ||
KnitLoRA
|
17.21 | 56.33 | 21.76 | 26.20 | ||
KnitLoRA
|
17.21 | 56.67 | 21.99 | 26.63 | ||
KnitLoRA
|
14.86 | 56.33 | 19.86 | 25.56 | ||
| LLaMA2 | 7B | LoRA3 | 9.02 | 37.50 | 9.55 | 16.08 |
| AdaLoRA5 | - | 5.92 | - | 10.39 | ||
| LoKr6 | - | 6.67 | - | 14.94 | ||
ResLoRA 4
|
11.83 | 45.39 | 13.57 | 16.95 | ||
KnitLoRA
|
14.94 | 54.28 | 19.03 | 21.71 | ||
KnitLoRA
|
13.87 | 50.00 | 17.74 | 20.50 | ||
KnitLoRA
|
10.39 | 47.70 | 11.07 | 17.69 |
Fig. 5.
Main Results of Fine-tuning Qwen2-1.5B on SVAMP Task. We present the evaluation accuracy curve and the comparison of best accuracy scores.
NLU performance
Table 4 presents NLU evaluation outcomes, demonstrating the competitiveness of our framework. The proposed method achieves the state-of-the-art (SOTA) performance across all tasks, showing consistent improvements over baseline approaches. Notably, all three KnitLoRA variants outperform existing methods on the WNLI task, further substantiating the architectural advantages in enhancing model adaptability for language understanding scenarios. These results validate the effectiveness of dense connectivity patterns in parameter-efficient fine-tuning frameworks for NLU tasks. At the same time, on the GLUE benchmark, KnitLoRA
consistently outperforms Full Fine-Tuning (Table 5), demonstrating that reducing the number of trainable parameters does not necessarily lead to performance degradation and highlighting the strong capability of the KnitLoRA architecture. On GSM8K, the performance gap between KnitLoRA and Full Fine-Tuning is very small (
). This result indicates that, in complex multi-step arithmetic reasoning scenarios, KnitLoRA approaches the performance upper bound of Full Fine-Tuning while maintaining parameter efficiency.
Table 4.
Main Results of Fine-tuning RoBERTa-large on GLUE. We implement the code for all structures using the publicly available Huggingface Transformers36 library and PyTorch37. For other baseline methods, we implement them through Huggingface’s PEFT38 package, for ResLoRA
, we use its open-source code. We used two L20 GPU to complete all the experiments.
| Method | Conf. | CoLA | MRPC | RTE | STS-B | WNLI | SST-2 | QNLI |
|---|---|---|---|---|---|---|---|---|
| LoRA3 | ICLR’22 | 66.02 | 90.69 | 84.48 | 91.72 | 54.93 | 95.99 | 94.55 |
| AdaLoRA5 | ICLR’23 | 66.59 | 88.97 | 55.60 | 91.53 | 54.93 | 96.09 | 94.01 |
| LoKr6 | ICLR’24 | 57.11 | 84.80 | 69.31 | 90.16 | 50.70 | 95.64 | 93.74 |
ResLoRA 4
|
ACL’24 | 64.39 | 90.11 | 84.50 | 91.56 | 54.93 | 94.28 | 93.26 |
DoRA 35
|
ACL’24 | 65.35 | 90.05 | 81.73 | 91.34 | – | 95.21 | 92.97 |
DoRA 35
|
ACL’24 | 64.90 | 89.72 | 79.15 | 91.28 | – | 94.98 | 92.93 |
| MTL-LoRA22 | AAAI’25 | 66.12 | 90.33 | 84.06 | 91.73 | 55.47 | 95.38 | 94.15 |
KnitLoRA (Ours) |
– | 64.40 | 89.95 | 81.23 | 90.91 | 56.34 | 94.84 | 93.89 |
KnitLoRA (Ours) |
– | 67.25 | 91.18 | 84.84 | 91.47 | 56.34 | 96.45 | 94.62 |
KnitLoRA (Ours) |
– | 67.08 | 89.95 | 85.92 | 91.75 | 56.34 | 96.10 | 93.94 |
Table 5.
Comparison between KnitLoRA
and FFT. On the GLUE benchmark, KnitLoRA
consistently outperforms FFT. Bold values indicate better results.
| Model | Benchmark | Task | KnitLoRA
|
FFT |
|---|---|---|---|---|
| LLaMA3.2 | GSM8K | - | 17.21 | 17.82 |
| RoBERTa-large | GLUE | CoLA | 67.25 | 59.43 |
| MRPC | 91.18 | 68.38 | ||
| RTE | 84.84 | 71.84 | ||
| STS-B | 91.47 | 89.38 | ||
| WNLI | 56.34 | 56.34 | ||
| SST-2 | 96.45 | 93.92 | ||
| QNLI | 94.62 | 92.04 |
T2I performance
Figure 6 demonstrates comparative outcomes of the text-to-image generation experiment. We evaluated performance for LoRA, ResLoRA
, and KnitLoRA
across two distinct prompts. In Figure 6b, at training steps 240 and 300, conventional methods (LoRA/ResLoRA) still produce photorealistic foxes, whereas KnitLoRA generates Pokémon-style fox characters. For the rabbit generation task (Figure 6c), outputs from KnitLoRA at identical steps exhibit stronger cartoon stylization compared to baseline approaches. Similar advantages of the KnitLoRA can be observed in Figures 6d) and 6e). These comparisons confirm KnitLoRA’s superior capability in capturing domain-specific visual characteristics. This further demonstrates the superiority of deploying our architecture in multimodal domains. From the results, it can be concluded that whether in NLP experiments or text-to-image experiments, KnitLoRA has been confirmed to enable deeper semantic reasoning of models.
Fig. 6.
Results of text-to-image task. We compare images generated by LoRA, ResLoRA, and KnitLoRA.
Ablation studies
The rank of LoRA
We conducted ablation studies on KnitLoRA
with varying rank configurations to analyze their impact on model performance, as shown in Table 6. Consistent with observations in3, our results confirm that optimal performance occurs at lower ranks, with progressive performance degradation as rank increases. A marginal improvement emerges when the rank is greater than 16, though such high-rank configurations are impractical due to elevated computational demands. The lack of meaningful subspace enhancement with higher ranks validates the sufficiency of low-rank parameterization in our framework, demonstrating that low-rank adaptation matrices retain full representational capacity without introducing redundant parameters.
Table 6.
Results of KnitLoRA with varying ranks on SVAMP. Rank selection follows the experiment of LoRA3. We report the best accuracy for each rank.
| Rank | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
|---|---|---|---|---|---|---|---|
| Accuracy (%) | 57.67 | 57.00 | 56.33 | 56.33 | 57.67 | 57.67 | 57.67 |
Merge approach
While our parameter merging strategy offers computational advantages during inference, it remains critical to evaluate its impact on model accuracy. As demonstrated in Table 7, direct deployment of unmerged post-training weights fails to generate valid outputs due to structural inconsistencies introduced by dense connectivity paths, which create inconsistencies between training and inference phases. This confirms the necessity of the merging procedure. Between the two proposed methods, DP-Merge achieves superior accuracy compared to DI-Merge, even outperforming unmerged inference with dense paths by 0.34%, demonstrating that proper merging preserves or enhances model performance. DI-Merge estimates the scaling factor 
by computing the ratio between the Frobenius norm of the input tensor and the averaged Frobenius norm of the summed inputs from preceding layers. This approach implicitly assumes a stable linear proportionality between them. However, when the magnitude of the input tensor varies substantially, the averaged norm may fail to accurately capture the dynamic fluctuations of the input, thereby introducing approximation errors. For example, in long-tailed distribution scenarios, the input magnitudes of low-frequency samples (e.g., rare medical terms, archaic legal jargon, rare dialects, or specific technical nomenclature) differ significantly from those of high-frequency samples. Consequently, the 
computed by DI-Merge based on a global average norm cannot precisely adapt to the features of these low-frequency samples. In contrast, DP-Merge computes the scaling factor based on weight norms, which are independent of input fluctuations, resulting in more stable performance on long-tailed samples compared to DI-Merge. These quantitative analyses validate the effectiveness of our merging strategies in maintaining accuracy while eliminating inference overhead.
Table 7.
Results of KnitLoRAin using different merging methods on SVAMP. Mergeno means directly using trained weights without merging/dense connection; w/o Merge uses dense connection without merging; DI-Merge/DP-Merge are described in section 3.3. (
, best accuracies reported.).
| Method | Mergeno | DI-Merge | LoRA | w/o Merge | DP-Merge |
|---|---|---|---|---|---|
| Accuracy (%) | 0.33 | 54.00 | 54.67 | 56.33 | 56.67 |
The number of previous KnitLoRA blocks
We analyzed the effect of connection density by adjusting the number of preceding LoRA blocks in KnitLoRA
and KnitLoRA
, as summarized in Table 8. Except for KnitLoRA
on SVAMP, optimal performance consistently occurs at moderate 
values rather than maximal connectivity configurations, indicating excessive connection density degrades effectiveness. This suggests moderate 
settings balance information flow and structural complexity more effectively. Furthermore, multi-block connectivity demonstrates practical benefits by enabling gradient propagation through alternative pathways, allowing gradients to bypass potential degradation points in deep architectures. Based on the experimental results in Table 8, we propose a guideline for selecting the 
parameter. For new tasks, we recommend using 
as the initial configuration. This setting achieves near-optimal performance on core tasks such as SVAMP and GSM8K, while effectively balancing cross-layer feature reuse and training stability, making it suitable for most downstream scenarios. When computational resources are limited, configurations with 
are preferred. This setting minimizes GPU memory consumption during training while still maintaining performance superior to the original LoRA.
Table 8.
Results with Different Numbers of Previous LoRA Blocks in KnitLoRA
and KnitLoRA
. ‘
’ means we have connected all preceding LoRA blocks; ‘0’ means we use the original LoRA method.
| Previous Blocks | KnitLoRA
|
KnitLoRA
|
||
|---|---|---|---|---|
| SVAMP | GSM8K | SVAMP | GSM8K | |
| 0 | 54.67 | 15.85 | 54.67 | 15.85 |
| 1 | 55.67 | 16.91 | 54.67 | 17.44 |
| 2 | 55.67 | 16.07 | 54.67 | 15.69 |
| 3 | 55.67 | 17.44 | 55.00 | 15.92 |
| 4 | 58.33 | 17.21 | 55.67 | 15.92 |
| 5 | 56.67 | 17.51 | 56.00 | 15.92 |
 |
56.67 | 17.21 | 56.33 | 14.86 |
Impact of different KnitLoRA structures on efficiency
Beyond the raw efficiency metrics reported in Table 9, the observed differences among KnitLoRA structures can be further explained by their architectural characteristics. KnitLoRA
introduces dense connections only at the input aggregation stage, which requires minimal additional tensor operations and does not increase the number of trainable parameters. Consequently, its GPU memory consumption and throughput remain close to those of standard LoRA. This design provides an effective trade-off between computational overhead and representation enhancement, making KnitLoRA
particularly suitable for large-scale training scenarios where efficiency is a critical constraint. In contrast, KnitLoRA
and KnitLoRA
perform dense aggregation directly on the low-rank matrices across multiple LoRA blocks. While this design slightly increases intermediate tensor operations and memory usage during training, it substantially improves cross-layer feature interaction and parameter expressiveness. In particular, the B-Dense structure enables the final projection matrix to integrate information from multiple historical LoRA blocks, effectively increasing the expressive capacity of the low-rank adaptation without increasing the rank. This explains why KnitLoRA
and KnitLoRA
often achieve higher accuracy and faster convergence in complex reasoning tasks despite their higher training cost. More importantly, all KnitLoRA variants maintain the core efficiency advantage of LoRA during inference. Through the proposed parameter merging strategies described in Section 3.3, the dense connections introduced during training can be eliminated and converted into standard LoRA weight updates. As a result, KnitLoRA incurs no additional inference latency or memory overhead compared with the original LoRA during inference. Therefore, KnitLoRA achieves a desirable balance: richer cross-layer feature utilization during training and identical deployment efficiency during inference.
Table 9.
Performance comparison of four LoRA variants in model training. This table summarizes the core training efficiency and GPU memory consumption metrics of four LoRA-based training methods. All experiments fine-tune the LLaMA3.2-1B model on the SVAMP task with identical hyperparameters and 1720 global training steps. The unit of measurement for throughput is tokens per second.
| Method | Time Cost (s) | Tokens per Global Step | Throughput (Input+Output) | Throughput (Input Only) | GPU Memory Allocated (GB) |
|---|---|---|---|---|---|
| LoRA | 3360 | 16000 | 8190.48 | 4095.24 | 6.37 |
KnitLoRA
|
3780 | 16000 | 7280.42 | 3640.21 | 6.42 |
KnitLoRA
|
5460 | 16000 | 5040.29 | 2520.15 | 10.71 |
KnitLoRA
|
7860 | 16000 | 3501.27 | 1750.64 | 10.24 |
Experimental analysis
Dense connections benefit the training phase
While theoretical analysis in section 3.4 establishes that dense connectivity mitigates gradient vanishing and accelerates training, empirical validation remains critical. Figure 7 compares training loss dynamics between standard LoRA and KnitLoRA
under varying 
settings. The baseline LoRA (equivalent to KnitLoRA
with 
) exhibits the slowest loss reduction, whereas increasing preceding block connections consistently improves convergence speed and final loss minimization. This validates that dense pathways enhance gradient propagation efficiency, particularly in deep networks.
Fig. 7.
Training loss for KnitLoRA
with different 
values on SVAMP. 
= ’
’ indicates that we have connected all preceding LoRA blocks. LoRA can be regarded as 
= 0.
Dense connections lead to better model inference performance
Our architecture demonstrates superior inference accuracy alongside training benefits. To investigate this correlation, we compared training loss dynamics across LoRA variants as shown in Figure 8. The results reveal KnitLoRA achieves faster loss reduction and converges to lower final values compared to LoRA and ResLoRA. This indicates enhanced optimization efficiency during training, enabling the model to extract greater knowledge from identical datasets. We attribute the improved semantic reasoning performance to this thorough training mechanism, where dense connectivity facilitate more effective parameter updates and feature learning throughout the adaptation process. Quantitative metrics confirm this architectural advantage translates consistently into higher evaluation accuracy across diverse task domains.
Fig. 8.
Training loss for fine-tuning Qwen2-1.5B on SVAMP. We compared the training loss of three different structures: LoRA, ResLoRA
, and KnitLoRA
.
Conclusion and future work
In this study, our work reveals the limitations of the traditional LoRA method using a single-path structure and highlights the advantages brought by dense connections. To this end, we propose a novel low-rank adaptation framework–KnitLoRA, consisting of three structures that introduce dense connections during training and remove these paths during inference through merging strategies. We also propose HSCoT, a 1.5K-sample benchmark drawn from HellaSwag and enhanced with Chain-of-Thought annotations, which can effectively evaluate a model’s ability to perform multi-step reasoning under the CoT paradigm. We conducted experiments on various tasks and models, which confirmed the effectiveness of our approach. Meanwhile, our architecture also demonstrated powerful multimodal learning capabilities in experiments. The advantages of our method can be summarized as follows: KnitLoRA offers significantly enhanced semantic reasoning capabilities for models with fewer training steps and no additional trainable parameters compared to baseline methods. Future work will focus on improving the robustness of the merging strategy and optimizing the training efficiency of dense connectivity, particularly under long-tailed data distributions where rare samples and highly variable input magnitudes may amplify approximation errors during merging and increase optimization instability.
Author contributions
H.Q. conceived and conducted the main experiments, developed the core codebase, and drafted the manuscript. Y.N. designed the study framework, wrote major portions of the manuscript, and prepared Figures 1-2. J.L. and X.Z. performed the validation experiments and data analysis for the comparative study section. Y.G. designed the overall system architecture and coordinated the project workflow. L.Y. proposed the original idea, led the technical design, and supervised the research. F.G., J.Z., F.F., Z.X., Q.W., and P.L. participated in manuscript revision, data curation, and results interpretation. All authors reviewed and approved the final manuscript.
Funding
This project is funded by the Zhejiang Provincial Natural Science Foundation of China (NO. LQN25F020019), the Fundamental Research Funds for the Provincial Universities of Zhejiang (NO. GK249909299001-033), and the Key Laboratory of Data Science and Intelligence Education (Hainan Normal University), Ministry of Education (NO. DSIE202403).
Data availability
The data used in this study are publicly available. Furthermore, the complete set of results data generated during this research, including trained model weights and configurations, will be made publicly available upon publication.
Declarations
Competing interests
The authors declare no competing interests.
Footnotes
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Contributor Information
Liqi Yan, Email: lqyan18@fudan.edu.cn.
Pan Li, Email: lipan@ieee.org.
References
- 1.Houlsby, N. et al. Parameter-Efficient Transfer Learning for NLP, In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Proceedings of Machine Learning Research, vol. 97, ed. by Chaudhuri, K., Salakhutdinov, R., (PMLR, 2019), pp. 2790–2799. http://proceedings.mlr.press/v97/houlsby19a.html
- 2.Li, X.L. & Liang, P. Prefix-Tuning: Optimizing Continuous Prompts for Generation, In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, ed. by Zong, C., Xia, F., Li, W., Navigli R. (Association for Computational Linguistics, 2021), pp. 4582–4597. 10.18653/V1/2021.ACL-LONG.353.(2021)
- 3.Hu, E.J. et al. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 (OpenReview.net, 2022). https://openreview.net/forum?id=nZeVKeeFYf9(2022)
- 4.Shi, S. et al. ResLoRA: Identity Residual Mapping in Low-RankAdaption. In Findings of the Association for ComputationalLinguistics: ACL 2024, ed. by Ku, L.W., Martins, A., Srikumar V., (Associationfor Computational Linguistics, Bangkok, Thailand, 2024), pp. 8870–8884.10.18653/v1/2024.findings-acl.525.https://aclanthology.org/2024.findings-acl.525/(2024)
- 5.Zhang, Q. et al. Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 (OpenReview.net, 2023). https://openreview.net/forum?id=lq62uWRJjiY(2023)
- 6.Yeh, S. et al. Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation.In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 (OpenReview.net, 2024). https://openreview.net/forum?id=wfzXa8e783(20214)
- 7.Patel, A., Bhattamishra, S., Goyal, N., Are NLP Models really able to Solve Simple Math Word Problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ed. by Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y., (Association for Computational Linguistics, Online,2021), pp. 2080–2094. 10.18653/v1/2021.naacl-main.168. https://aclanthology.org/2021.naacl-main.168/(2021)
- 8.Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A. & Choi, Y. HellaSwag: Can a Machine Really Finish Your Sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ed. by Korhonen, A., Traum, D., Màrquez, L. (Association for Computational Linguistics, Florence, Italy, 2019), pp. 4791–4800. 10.18653/v1/P19-1472. https://aclanthology.org/P19-1472/(2019)
- 9.Guo, D. et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)
- 10.Zaken, E.B., Goldberg, Y. & Ravfogel, S., BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, ed. by Muresan, S., Nakov, P., Villavicencio, A., (Association for Computational Linguistics, 2022), pp. 1–9. 10.18653/V1/2022.ACL-SHORT.1. (2022)
- 11.Pan, Z., Wang, K., Li, G., He, F. & Lai, Y. Finediffusion: Scaling up diffusion models for fine-grained image generation with 10,000 classes. Appl. Intell.55(5), 309. (2025).
- 12.Tian, C., Shi, Z., Guo, Z., Li, L. & Xu, C. Hydralora: An asymmetric lora architecture for efficient fine-tuning. CoRR abs/2404.19245 (2024). 10.48550/ARXIV.2404.19245. arXiv:2404.19245
- 13.Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K.Q., Densely Connected Convolutional Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 2261–2269. 10.1109/CVPR.2017.243(2017)
- 14.Zhang, J., Zhang, Y., Jin, Y., Xu, J. & Xu, X. Mdu-net: Multi-scale densely connected u-net for biomedical image segmentation. Health Inf. Sci. Syst.11(1), 13. 10.1007/S13755-022-00204-9 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Parihar, A. S. & Java, A. Densely connected convolutional transformer for single image dehazing. J. Vis. Commun. Image Represent.90, 103722. 10.1016/J.JVCIR.2022.103722 (2023). [Google Scholar]
- 16.Zhu, C., Wang, L., Zhao, W. & Lian, H. Image classification based on tensor network densenet model. Appl. Intell.54(8), 6624–6636 (2024). [Google Scholar]
- 17.Mehta, N. K., Prasad, S. S., Saurav, S., Saini, R. & Singh, S. Three-dimensional densenet self-attention neural network for automatic detection of student’s engagement. Appl. Intell.52(12), 13803–13823 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.He, K., Zhang, X., Ren, S. & Sun, J. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 (IEEE Computer Society, 2016), pp. 770–778. 10.1109/CVPR.2016.90.(2016)
- 19.Ford, W. Numerical linear algebra with applications: Using MATLAB (Academic Press, 2014)
- 20.Li, X., Peng, S., Yada, S., Wakamiya, S. & Aramaki, E. Genkp: Generative knowledge prompts for enhancing large language models. Appl. Intell.55(6), 464 (2025). [Google Scholar]
- 21.Cao, X. & Liu, Y. Relmkg: Reasoning with pre-trained language models and knowledge graphs for complex question answering. Appl. Intell.53(10), 12032–12046 (2023). [Google Scholar]
- 22.Yang, Y. et al. Mtl-lora: Low-rank adaptation for multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39 , pp. 22010–22018 (2025)
- 23.Rasley, J., Rajbhandari, S., Ruwase, O. & He, Y. DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters, in KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, ed. by Gupta, R., Liu, Y., Tang, J., Prakash B.A. (ACM, 2020), pp. 3505–3506. 10.1145/3394486.3406703.(2020)
- 24.Dubey, A. et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)
- 25.Yang, A. et al. Qwen2 technical report . https://arxiv.org/abs/2407.10671(2024)
- 26.Touvron, H. et al., Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
- 27.Cobbe, K. et al. Training verifiers to solve math word problems. CoRR abs/2110.14168 (2021). https://arxiv.org/abs/2110.14168. arXiv:2110.14168
- 28.Yu, L. et al. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. In The Twelfth International Conference on Learning Representations (2024). https://openreview.net/forum?id=N8N0hgNDRt
- 29.Amini, A. et al. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), ed. by Burstein, J., Doran, C., Solorio T., (Association for Computational Linguistics, Minneapolis, Minnesota, 2019), pp. 2357–2367. 10.18653/v1/N19-1245. https://aclanthology.org/N19-1245/(2019)
- 30.Liu, Y. et al. Roberta: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019). arXiv:1907.11692
- 31.Wang, A. et al. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 (OpenReview.net, 2019). https://openreview.net/forum?id=rJ4km2R5t7
- 32.Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 (Springer, 2015), pp. 234–241(2015)
- 33.Rombach, R., Blattmann, A., Lorenz, D., Esser, P. & Ommer, B. High-Resolution Image Synthesis with Latent Diffusion Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 (IEEE, 2022), pp. 10674–10685. 10.1109/CVPR52688.2022.01042.(2022)
- 34.Linoyts. Tuxemon dataset. https://huggingface.co/datasets/linoyts/Tuxemon (2024). Accessed: 2025-02-07
- 35.Mao, Y. et al. DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ed. by L.W. Ku, A. Martins, V. Srikumar (Association for Computational Linguistics, Bangkok, Thailand, 2024), pp. 11662–11675. 10.18653/v1/2024.acl-long.626. https://aclanthology.org/2024.acl-long.626/(2024) [DOI] [PMC free article] [PubMed]
- 36.Wolf, T. et al. Huggingface’s transformers: State-of-the-art natural language processing. CoRR abs/1910.03771arXiv:1910.03771 (2019)
- 37.Paszke A. et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, ed. by H.M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E.B. Fox, R. Garnett (2019), pp. 8024–8035. https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html(2019)
- 38.Mangrulkar, S. et al. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft (2022)
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The data used in this study are publicly available. Furthermore, the complete set of results data generated during this research, including trained model weights and configurations, will be made publicly available upon publication.