Authors: Hananel Hazan and Yanbo Zhang and Benedikt Hartl and Michael Levin
Cite as: \cite{hazan2026a}
arXiv: https://arxiv.org/abs/2604.08749
Summary in three sentences
This paper investigates how much task-specific information can be represented by low-rank LoRA adapters over frozen random backbones. The arXiv abstract argues that low-rank adapters can recover much of fully trained performance across diverse benchmarks, suggesting that task signal can live in a much smaller subspace than the full parameter count. For this project, it supports the premise that a low-dimensional adapter search space may be expressive enough for some tasks.
Main contribution
The main contribution is LottaLoRA, a training paradigm where the backbone is random and frozen while LoRA adapters carry the task-specific learning signal.
Key equations and definitions
The project-relevant notions are LoRA rank, frozen random scaffold, learned scaling , and the interpretation of saturation rank as a proxy for task intrinsic dimensionality.
Empirical results
Every number here must name the section or figure it comes from.
If you cannot locate it in the paper, do not write it.
The arXiv abstract reports experiments across multiple benchmarks and architecture families, but no exact benchmark numbers are recorded here until the tables or figures are read directly.
Relation to our project
- what we reuse: the idea that low-rank adapter subspaces can carry task-specific signal.
- what we dispute: random-backbone results do not automatically imply that ES can efficiently discover good adapter directions from sparse rewards.
- what we must reproduce: a rank sensitivity sweep for the smoke test or first LLM task.
Criticism and limitations
The paper is about trained LoRA adapters, not ES perturbation estimators. It supports the plausibility of low-rank adaptation but does not answer the optimisation question.