Design
This experiment tests the next falsifier from the draft empirical sketch proof:
whether the plain EGGROLL and decomposed EGGROLL LLM paths still behave
reasonably once answer-token likelihood is replaced by sampled-answer scalar
reward.
There is still no solid theory note for the project, so this should be read as
an empirical gate, not as a settled-theory test. The falsifying outcome for the
current draft claim would be a clear plain-EGGROLL win under matched reward
budget. A flat reward result would mostly say that this first exact-answer
benchmark is too sparse.
Results
20260801-113246-pretrained_llm_exact_reward_smoke_jean_zay-6224172
commit: 6224172
| metric | value |
|---|---|
| both_success_rate | 0.3333333333333333 |
| decomposed_mean_accuracy_improvement | 0.015625 |
| decomposed_mean_best_accuracy | 0.4322916666666667 |
| decomposed_mean_final_accuracy | 0.4114583333333333 |
| decomposed_mean_final_parseable_rate | 0.9791666666666666 |
| decomposed_mean_finite_epoch_fraction | 1 |
| decomposed_mean_parseable_rate_improvement | 0.0625 |
| decomposed_mean_shaped_epoch_fraction | 0 |
| decomposed_median_final_accuracy | 0.40625 |
| decomposed_success_rate | 0.3333333333333333 |
| decomposed_win_rate | 0.16666666666666666 |
| eggroll_mean_accuracy_improvement | 0.010416666666666666 |
| eggroll_mean_best_accuracy | 0.4479166666666667 |
| eggroll_mean_final_accuracy | 0.40625 |
| eggroll_mean_final_parseable_rate | 0.9791666666666666 |
| eggroll_mean_finite_epoch_fraction | 1 |
| eggroll_mean_parseable_rate_improvement | 0.0625 |
| eggroll_mean_shaped_epoch_fraction | 0 |
| eggroll_median_final_accuracy | 0.40625 |
| eggroll_success_rate | 0.3333333333333333 |
| epochs | 8 |
| eval_samples | 32 |
| mean_accuracy_gap_decomposed_minus_eggroll | 0.005208333333333333 |
| mean_improvement_gap_decomposed_minus_eggroll | 0.005208333333333333 |
| mean_parseable_gap_decomposed_minus_eggroll | 0 |
| num_pairs | 6 |
| num_ranks | 2 |
| num_seeds | 3 |
| num_target_modules | 7 |
| population | 16 |
| rank_1_decomposed_success_rate | 0.3333333333333333 |
| rank_1_decomposed_win_rate | 0 |
| rank_1_eggroll_success_rate | 0.3333333333333333 |
| rank_1_mean_accuracy_gap_decomposed_minus_eggroll | 0 |
| rank_2_decomposed_success_rate | 0.3333333333333333 |
| rank_2_decomposed_win_rate | 0.3333333333333333 |
| rank_2_eggroll_success_rate | 0.3333333333333333 |
| rank_2_mean_accuracy_gap_decomposed_minus_eggroll | 0.010416666666666666 |
Interpretation
Run 20260801-113246-pretrained_llm_exact_reward_smoke_jean_zay-6224172 cleared
the basic reward-only plumbing gate. Both variants ran through sampled
generation, integer parsing, scalar correctness reward, ES update, and final
metrics without numerical failure.
The result does not give a strong decomposed win. Decomposed EGGROLL was only
slightly ahead on mean final correctness and mean improvement, while plain
EGGROLL had the better best-achieved correctness. The win pattern is weak and
rank-dependent, so the honest interpretation is parity with a small favorable
hint for decomposed EGGROLL, not evidence strong enough for a paper claim.
The boring alternative explanation is sampling noise and benchmark granularity:
the evaluation set is deliberately tiny, and correctness moves in coarse jumps.
This run does not rule that out.
The useful next step is either a larger generated-arithmetic exact-reward sweep
with the same code path, or a true GSM8K-subset smoke that streams the dataset
inside the job workdir and keeps the same sampled-answer numeric reward.