Design
Which solid notes this experiment tests, and how.
Results
20260803-121737-pretrained_llm_gsm8k_scale_grid_jean_zay-937ecc2
commit: 937ecc2
| metric | value |
|---|---|
| both_success_rate | 0.3333333333333333 |
| decomposed_grid_0_lr_direction | 0.01 |
| decomposed_grid_0_lr_magnitude | 0.1 |
| decomposed_grid_0_mean_accuracy_gap_decomposed_minus_eggroll | 0 |
| decomposed_grid_0_mean_final_accuracy | 0.024305556279917557 |
| decomposed_grid_0_success_rate | 0.3333333333333333 |
| decomposed_grid_0_win_rate | 0 |
| decomposed_grid_1_lr_direction | 0.01 |
| decomposed_grid_1_lr_magnitude | 0.25 |
| decomposed_grid_1_mean_accuracy_gap_decomposed_minus_eggroll | 0.006944444340964158 |
| decomposed_grid_1_mean_final_accuracy | 0.031250000620881714 |
| decomposed_grid_1_success_rate | 0.3333333333333333 |
| decomposed_grid_1_win_rate | 0.3333333333333333 |
| decomposed_grid_2_lr_direction | 0.01 |
| decomposed_grid_2_lr_magnitude | 0.4 |
| decomposed_grid_2_mean_accuracy_gap_decomposed_minus_eggroll | 0.0034722223257025084 |
| decomposed_grid_2_mean_final_accuracy | 0.027777778605620067 |
| decomposed_grid_2_success_rate | 0.3333333333333333 |
| decomposed_grid_2_win_rate | 0.3333333333333333 |
| decomposed_grid_3_lr_direction | 0.02 |
| decomposed_grid_3_lr_magnitude | 0.1 |
| decomposed_grid_3_mean_accuracy_gap_decomposed_minus_eggroll | 0 |
| decomposed_grid_3_mean_final_accuracy | 0.024305556279917557 |
| decomposed_grid_3_success_rate | 0.3333333333333333 |
| decomposed_grid_3_win_rate | 0 |
| decomposed_grid_4_lr_direction | 0.02 |
| decomposed_grid_4_lr_magnitude | 0.25 |
| decomposed_grid_4_mean_accuracy_gap_decomposed_minus_eggroll | 0.0034722223257025084 |
| decomposed_grid_4_mean_final_accuracy | 0.027777778605620067 |
| decomposed_grid_4_success_rate | 0.3333333333333333 |
| decomposed_grid_4_win_rate | 0.3333333333333333 |
| decomposed_grid_5_lr_direction | 0.02 |
| decomposed_grid_5_lr_magnitude | 0.4 |
| decomposed_grid_5_mean_accuracy_gap_decomposed_minus_eggroll | 0.0034722223257025084 |
| decomposed_grid_5_mean_final_accuracy | 0.027777778605620067 |
| decomposed_grid_5_success_rate | 0.3333333333333333 |
| decomposed_grid_5_win_rate | 0.3333333333333333 |
| decomposed_lr_direction | 0.02 |
| decomposed_lr_direction_0_mean_accuracy_gap_decomposed_minus_eggroll | 0.003472222222222222 |
| decomposed_lr_direction_0_mean_final_accuracy | 0.02777777850213978 |
| decomposed_lr_direction_0_success_rate | 0.3333333333333333 |
| decomposed_lr_direction_0_value | 0.01 |
| decomposed_lr_direction_0_win_rate | 0.2222222222222222 |
| decomposed_lr_direction_1_mean_accuracy_gap_decomposed_minus_eggroll | 0.002314814883801672 |
| decomposed_lr_direction_1_mean_final_accuracy | 0.02662037116371923 |
| decomposed_lr_direction_1_success_rate | 0.3333333333333333 |
| decomposed_lr_direction_1_value | 0.02 |
| decomposed_lr_direction_1_win_rate | 0.2222222222222222 |
| decomposed_lr_magnitude | 0.02 |
| decomposed_lr_magnitude_0_mean_accuracy_gap_decomposed_minus_eggroll | 0 |
| decomposed_lr_magnitude_0_mean_final_accuracy | 0.024305556279917557 |
| decomposed_lr_magnitude_0_success_rate | 0.3333333333333333 |
| decomposed_lr_magnitude_0_value | 0.1 |
| decomposed_lr_magnitude_0_win_rate | 0 |
| decomposed_lr_magnitude_1_mean_accuracy_gap_decomposed_minus_eggroll | 0.005208333333333333 |
| decomposed_lr_magnitude_1_mean_final_accuracy | 0.029513889613250893 |
| decomposed_lr_magnitude_1_success_rate | 0.3333333333333333 |
| decomposed_lr_magnitude_1_value | 0.25 |
| decomposed_lr_magnitude_1_win_rate | 0.3333333333333333 |
| decomposed_lr_magnitude_2_mean_accuracy_gap_decomposed_minus_eggroll | 0.0034722223257025084 |
| decomposed_lr_magnitude_2_mean_final_accuracy | 0.027777778605620067 |
| decomposed_lr_magnitude_2_success_rate | 0.3333333333333333 |
| decomposed_lr_magnitude_2_value | 0.4 |
| decomposed_lr_magnitude_2_win_rate | 0.3333333333333333 |
| decomposed_mean_accuracy_improvement | 0.009837963514857821 |
| decomposed_mean_best_accuracy | 0.031250000620881714 |
| decomposed_mean_final_accuracy | 0.027199074832929507 |
| decomposed_mean_final_greedy_accuracy | 0.027777778605620067 |
| decomposed_mean_final_greedy_parseable_rate | 1 |
| decomposed_mean_final_parseable_rate | 0.9924768805503845 |
| decomposed_mean_finite_epoch_fraction | 0 |
| decomposed_mean_greedy_accuracy_improvement | 0 |
| decomposed_mean_greedy_parseable_rate_improvement | 0 |
| decomposed_mean_parseable_rate_improvement | -0.0005786816279093424 |
| decomposed_mean_shaped_epoch_fraction | 0.16666666666666666 |
| decomposed_mean_train_accuracy | 0.021267361111111112 |
| decomposed_mean_train_parseable_rate | 0.9891493055555556 |
| decomposed_median_final_accuracy | 0.02083333395421505 |
| decomposed_success_rate | 0.3333333333333333 |
| decomposed_tangent_project_direction | 0 |
| decomposed_win_rate | 0.2222222222222222 |
| eggroll_mean_accuracy_improvement | 0.006944444961845875 |
| eggroll_mean_best_accuracy | 0.031250000620881714 |
| eggroll_mean_final_accuracy | 0.024305556279917557 |
| eggroll_mean_final_greedy_accuracy | 0.027777778605620067 |
| eggroll_mean_final_greedy_parseable_rate | 1 |
| eggroll_mean_final_parseable_rate | 0.9930555820465088 |
| eggroll_mean_finite_epoch_fraction | 0 |
| eggroll_mean_greedy_accuracy_improvement | 0 |
| eggroll_mean_greedy_parseable_rate_improvement | 0 |
| eggroll_mean_parseable_rate_improvement | 1.9868214925130207e-08 |
| eggroll_mean_shaped_epoch_fraction | 0.4166666666666667 |
| eggroll_mean_train_accuracy | 0.009114583333333334 |
| eggroll_mean_train_parseable_rate | 0.984375 |
| eggroll_median_final_accuracy | 0.02083333395421505 |
| eggroll_success_rate | 0.3333333333333333 |
| epochs | 4 |
| eval_interval | 0 |
| eval_samples | 96 |
| generations_per_prompt | 8 |
| mean_accuracy_gap_decomposed_minus_eggroll | 0.0028935185530119473 |
| mean_greedy_accuracy_gap_decomposed_minus_eggroll | 0 |
| mean_greedy_improvement_gap_decomposed_minus_eggroll | 0 |
| mean_improvement_gap_decomposed_minus_eggroll | 0.0028935185530119473 |
| mean_parseable_gap_decomposed_minus_eggroll | -0.0005787014961242676 |
| num_decomposed_lr_directions | 2 |
| num_decomposed_lr_magnitudes | 3 |
| num_pairs | 18 |
| num_ranks | 1 |
| num_seeds | 3 |
| num_target_modules | 7 |
| population | 64 |
| prompts_per_epoch | 8 |
| rank_1_decomposed_success_rate | 0.3333333333333333 |
| rank_1_decomposed_win_rate | 0.2222222222222222 |
| rank_1_eggroll_success_rate | 0.3333333333333333 |
| rank_1_mean_accuracy_gap_decomposed_minus_eggroll | 0.0028935185530119473 |
| rank_1_mean_greedy_accuracy_gap_decomposed_minus_eggroll | 0 |
| wall_clock_seconds | 10266.024114018306 |
Interpretation
Run 20260803-121737-pretrained_llm_gsm8k_scale_grid_jean_zay-937ecc2 is a small positive signal for scale-calibrated decomposed EGGROLL, but not a decisive behavioural win. The best sampled-accuracy setting used the lower direction LR with an intermediate magnitude LR, matching the scale-diagnostic suspicion that the old magnitude LR was too small. Greedy accuracy stayed tied with plain EGGROLL, and overall exact-answer accuracy remains very low, so the result should guide the next scale choice rather than support a strong claim.