Design
This follow-up to the grouped GSM8K reward sweep tests whether decomposed EGGROLL was being hurt by magnitude-step scale. It keeps the grouped exact-answer reward setup and plain EGGROLL comparison fixed, then sweeps decomposed magnitude learning rates while reusing the same direction learning rate.
Results
20260803-011634-pretrained_llm_gsm8k_grouped_magnitude_lr_sweep_jean_zay-b650945
commit: b650945
| metric | value |
|---|---|
| both_success_rate | 0.3333333333333333 |
| decomposed_lr_direction | 0.02 |
| decomposed_lr_magnitude | 0.02 |
| decomposed_lr_magnitude_0_mean_accuracy_gap_decomposed_minus_eggroll | -0.005208333333333333 |
| decomposed_lr_magnitude_0_mean_final_accuracy | 0.024305556279917557 |
| decomposed_lr_magnitude_0_success_rate | 0.3333333333333333 |
| decomposed_lr_magnitude_0_value | 0 |
| decomposed_lr_magnitude_0_win_rate | 0 |
| decomposed_lr_magnitude_1_mean_accuracy_gap_decomposed_minus_eggroll | -0.005208333333333333 |
| decomposed_lr_magnitude_1_mean_final_accuracy | 0.024305556279917557 |
| decomposed_lr_magnitude_1_success_rate | 0.3333333333333333 |
| decomposed_lr_magnitude_1_value | 0.005 |
| decomposed_lr_magnitude_1_win_rate | 0 |
| decomposed_lr_magnitude_2_mean_accuracy_gap_decomposed_minus_eggroll | -0.003472222170482079 |
| decomposed_lr_magnitude_2_mean_final_accuracy | 0.026041667442768812 |
| decomposed_lr_magnitude_2_success_rate | 0.3333333333333333 |
| decomposed_lr_magnitude_2_value | 0.01 |
| decomposed_lr_magnitude_2_win_rate | 0 |
| decomposed_lr_magnitude_3_mean_accuracy_gap_decomposed_minus_eggroll | -0.003472222170482079 |
| decomposed_lr_magnitude_3_mean_final_accuracy | 0.026041667442768812 |
| decomposed_lr_magnitude_3_success_rate | 0.3333333333333333 |
| decomposed_lr_magnitude_3_value | 0.015 |
| decomposed_lr_magnitude_3_win_rate | 0 |
| decomposed_lr_magnitude_4_mean_accuracy_gap_decomposed_minus_eggroll | -0.001736111007630825 |
| decomposed_lr_magnitude_4_mean_final_accuracy | 0.027777778605620067 |
| decomposed_lr_magnitude_4_success_rate | 0.3333333333333333 |
| decomposed_lr_magnitude_4_value | 0.1 |
| decomposed_lr_magnitude_4_win_rate | 0 |
| decomposed_mean_accuracy_improvement | 0.008333333892126878 |
| decomposed_mean_best_accuracy | 0.03645833395421505 |
| decomposed_mean_final_accuracy | 0.02569444521019856 |
| decomposed_mean_final_greedy_accuracy | 0.027777778605620067 |
| decomposed_mean_final_greedy_parseable_rate | 0.9947916865348816 |
| decomposed_mean_final_parseable_rate | 0.9777777850627899 |
| decomposed_mean_finite_epoch_fraction | 1 |
| decomposed_mean_greedy_accuracy_improvement | 0 |
| decomposed_mean_greedy_parseable_rate_improvement | -0.0017361044883728027 |
| decomposed_mean_parseable_rate_improvement | 0.005555536349614461 |
| decomposed_mean_shaped_epoch_fraction | 0.375 |
| decomposed_mean_train_accuracy | 0.016276041666666668 |
| decomposed_mean_train_parseable_rate | 0.9697265625 |
| decomposed_median_final_accuracy | 0.02083333395421505 |
| decomposed_success_rate | 0.3333333333333333 |
| decomposed_tangent_project_direction | 0 |
| decomposed_win_rate | 0 |
| eggroll_mean_accuracy_improvement | 0.012152778295179209 |
| eggroll_mean_best_accuracy | 0.038194445272286735 |
| eggroll_mean_final_accuracy | 0.029513889613250893 |
| eggroll_mean_final_greedy_accuracy | 0.027777778605620067 |
| eggroll_mean_final_greedy_parseable_rate | 0.9965277910232544 |
| eggroll_mean_final_parseable_rate | 0.9809027910232544 |
| eggroll_mean_finite_epoch_fraction | 1 |
| eggroll_mean_greedy_accuracy_improvement | 0 |
| eggroll_mean_greedy_parseable_rate_improvement | 0 |
| eggroll_mean_parseable_rate_improvement | 0.00868054231007894 |
| eggroll_mean_shaped_epoch_fraction | 0.4583333333333333 |
| eggroll_mean_train_accuracy | 0.013997395833333334 |
| eggroll_mean_train_parseable_rate | 0.9710286458333334 |
| eggroll_median_final_accuracy | 0.026041666977107525 |
| eggroll_success_rate | 0.3333333333333333 |
| epochs | 8 |
| eval_samples | 96 |
| generations_per_prompt | 8 |
| mean_accuracy_gap_decomposed_minus_eggroll | -0.00381944440305233 |
| mean_greedy_accuracy_gap_decomposed_minus_eggroll | 0 |
| mean_greedy_improvement_gap_decomposed_minus_eggroll | 0 |
| mean_improvement_gap_decomposed_minus_eggroll | -0.00381944440305233 |
| mean_parseable_gap_decomposed_minus_eggroll | -0.0031250059604644776 |
| num_decomposed_lr_magnitudes | 5 |
| num_pairs | 30 |
| num_ranks | 2 |
| num_seeds | 3 |
| num_target_modules | 7 |
| population | 64 |
| prompts_per_epoch | 8 |
| rank_1_decomposed_success_rate | 0.3333333333333333 |
| rank_1_decomposed_win_rate | 0 |
| rank_1_eggroll_success_rate | 0.3333333333333333 |
| rank_1_mean_accuracy_gap_decomposed_minus_eggroll | -0.0027777778605620066 |
| rank_1_mean_greedy_accuracy_gap_decomposed_minus_eggroll | 0 |
| rank_2_decomposed_success_rate | 0.3333333333333333 |
| rank_2_decomposed_win_rate | 0 |
| rank_2_eggroll_success_rate | 0.3333333333333333 |
| rank_2_mean_accuracy_gap_decomposed_minus_eggroll | -0.004861110945542654 |
| rank_2_mean_greedy_accuracy_gap_decomposed_minus_eggroll | 0 |
| wall_clock_seconds | 2143.903599434532 |
Interpretation
Run 20260803-011634-pretrained_llm_gsm8k_grouped_magnitude_lr_sweep_jean_zay-b650945 does not support the scale-control explanation. All variants stayed finite, but decomposed EGGROLL remained behind plain EGGROLL on sampled final accuracy, while greedy accuracy was tied. The largest magnitude learning rate was the least bad decomposed setting, but it still did not produce a decomposed win. Treat this as another negative result for decomposed EGGROLL on this sparse GSM8K exact-answer setup.