Design
This page merges the matrix-only experiments that validated the ES plumbing before any LLM benchmark. These runs test controlled low-rank matrix learning on Jean Zay. They are implementation and geometry checks, not evidence about language-task learning.
The sequence was:
- plain EGGROLL matrix-learning smoke;
- single-seed plain-vs-decomposed comparison;
- seed/rank robustness sweep with full, direction-only, and magnitude-only decomposed ablations.
Results
20260731-115158-eggroll_matrix_learning_jean_zay-5efd8d0
commit: 5efd8d0
| metric | value |
|---|---|
| initial_loss | 2.9565727710723877 |
| final_loss | 0.203643798828125 |
| best_loss | 0.203643798828125 |
| finite_epoch_fraction | 1 |
| success | 1 |
20260731-140705-eggroll_vs_decomposed_matrix_learning_jean_zay-05ce937
commit: 05ce937
| metric | value |
|---|---|
| eggroll_final_loss | 0.203643798828125 |
| decomposed_final_loss | 0.09557401388883591 |
| loss_gap_decomposed_minus_eggroll | -0.10806978493928909 |
| eggroll_success | 1 |
| decomposed_success | 1 |
| decomposed_wins | 1 |
20260731-162711-matrix_robustness_sweep_jean_zay-04fee3e
commit: 04fee3e
| metric | value |
|---|---|
| num_seeds | 10 |
| num_ranks | 4 |
| num_records | 160 |
| eggroll_success_rate | 0.65 |
| dora_full_success_rate | 1 |
| dora_direction_only_success_rate | 0 |
| dora_magnitude_only_success_rate | 0 |
| eggroll_mean_final_loss | 1.91025531408377 |
| dora_full_mean_final_loss | 0.39386328402906656 |
| dora_direction_only_mean_final_loss | 4.286366328597069 |
| dora_magnitude_only_mean_final_loss | 4.3086372256278995 |
| dora_full_win_rate_vs_eggroll | 0.8 |
| dora_full_mean_loss_gap_vs_eggroll | -1.5163920300547034 |
Interpretation
The matrix sequence did what it was meant to do. Plain EGGROLL learned the toy low-rank matrix objective, which made it credible as the baseline implementation. The decomposed DoRA-style variant then beat plain EGGROLL in the single-seed comparison and, more importantly, in most paired comparisons in the seed/rank sweep.
The ablations are important: direction-only and magnitude-only decomposed variants both failed on the sweep, while the full decomposed method succeeded in every tested condition. That supports the specific magnitude-plus-direction story rather than a generic “more parameters” story.
This evidence is still synthetic and matrix-only. It justifies moving to LLM smoke tests and then to a real reward benchmark; it is not a final project claim.