Authors: Shih-Yang Liu and Chien-Yi Wang and Hongxu Yin and Pavlo Molchanov and Yu-Chiang Frank Wang and Kwang-Ting Cheng and Min-Hung Chen
Cite as: \cite{liu2024dora}
arXiv: https://arxiv.org/abs/2402.09353
Summary in three sentences
DoRA studies why LoRA can lag behind full fine-tuning and argues that one missing ingredient is an explicit decomposition of weight magnitude and direction. The method learns magnitude separately while using a LoRA-style low-rank update for directional changes. For this project, DoRA is the parameterisation template for making ES search less entangled than direct LoRA-space perturbation.
Main contribution
The main contribution is Weight-Decomposed Low-Rank Adaptation: a PEFT method that decomposes pretrained weights into magnitude and direction, then applies low-rank adaptation to the direction while preserving no additional inference overhead after merging.
Key equations and definitions
The relevant definition for this project is the decomposition of a pretrained weight into magnitude and direction, with low-rank adaptation applied to the directional component. The exact implementation details need to be checked against the paper before we reproduce the method in code.
Empirical results
Every number here must name the section or figure it comes from.
If you cannot locate it in the paper, do not write it.
The arXiv abstract reports improvements over LoRA on fine-tuning LLaMA, LLaVA, and VL-BART across commonsense reasoning, visual instruction tuning, and image/video-text understanding. No project note numbers are recorded here until the relevant tables or figures are read directly.
Relation to our project
- what we reuse: the magnitude/direction decomposition as the candidate search space for ES.
- what we dispute: not the DoRA result itself, but the assumption that gradient-based PEFT is the only natural use of the decomposition.
- what we must reproduce: a minimal DoRA-style adapter parameterisation with matched parameter and compute accounting against LoRA.
Criticism and limitations
DoRA is designed as a gradient-based PEFT method, not as an ES search space. The project therefore needs to test whether the decomposition still helps when updates come from reward-weighted perturbations rather than backpropagation.