Design
Draft-guided diagnostic for the Qwen2.5-0.5B GSM8K reward harness. No project theory note is solid yet, so this run does not test a settled claim. It checks whether reward-driven ES updates are destructive by logging per-epoch train accuracy, eval accuracy, parseability, reward variance, prompt-logit KL drift from the initial model, and effective update norms for plain EGGROLL and decomposed EGGROLL.
Results
20260809-165410-qwen05_gsm8k_destructive_update_diagnostic_jean_zay-4b5ea4f
commit: 4b5ea4f
| metric | value |
|---|---|
| decomposed_lr_direction | 0.01 |
| decomposed_lr_magnitude_0_mean_eval_accuracy_change | -0.0625 |
| decomposed_lr_magnitude_0_mean_final_eval_accuracy | 0.3333333432674408 |
| decomposed_lr_magnitude_0_mean_final_eval_prompt_kl | 0.2183033674955368 |
| decomposed_lr_magnitude_0_mean_final_train_accuracy | 0.375 |
| decomposed_lr_magnitude_0_mean_initial_eval_accuracy | 0.3958333432674408 |
| decomposed_lr_magnitude_0_mean_initial_train_accuracy | 0.25 |
| decomposed_lr_magnitude_0_mean_max_eval_prompt_kl | 0.2183033674955368 |
| decomposed_lr_magnitude_0_mean_mean_fitness_std | 0.9011701792478561 |
| decomposed_lr_magnitude_0_mean_mean_weight_relative_update | 0.008527024193778446 |
| decomposed_lr_magnitude_0_mean_train_accuracy_change | 0.125 |
| decomposed_lr_magnitude_0_value | 0.25 |
| decomposed_lr_magnitude_1_mean_eval_accuracy_change | -0.0625 |
| decomposed_lr_magnitude_1_mean_final_eval_accuracy | 0.3333333432674408 |
| decomposed_lr_magnitude_1_mean_final_eval_prompt_kl | 0.46276527643203735 |
| decomposed_lr_magnitude_1_mean_final_train_accuracy | 0.25 |
| decomposed_lr_magnitude_1_mean_initial_eval_accuracy | 0.3958333432674408 |
| decomposed_lr_magnitude_1_mean_initial_train_accuracy | 0.25 |
| decomposed_lr_magnitude_1_mean_max_eval_prompt_kl | 0.46276527643203735 |
| decomposed_lr_magnitude_1_mean_mean_fitness_std | 0.9156333953142166 |
| decomposed_lr_magnitude_1_mean_mean_weight_relative_update | 0.011528683619549757 |
| decomposed_lr_magnitude_1_mean_train_accuracy_change | 0 |
| decomposed_lr_magnitude_1_value | 0.35 |
| decomposed_mean_eval_accuracy_change | -0.0625 |
| decomposed_mean_final_eval_accuracy | 0.3333333432674408 |
| decomposed_mean_final_eval_prompt_kl | 0.3405343219637871 |
| decomposed_mean_final_train_accuracy | 0.3125 |
| decomposed_mean_initial_eval_accuracy | 0.3958333432674408 |
| decomposed_mean_initial_train_accuracy | 0.25 |
| decomposed_mean_max_eval_prompt_kl | 0.3405343219637871 |
| decomposed_mean_mean_fitness_std | 0.9084017872810364 |
| decomposed_mean_mean_weight_relative_update | 0.010027853906664101 |
| decomposed_mean_train_accuracy_change | 0.0625 |
| decomposed_sigma_direction | 0.01 |
| decomposed_sigma_magnitude | 0.02 |
| decomposed_tangent_project_direction | 0 |
| eggroll_lr | 0.02 |
| eggroll_mean_eval_accuracy_change | -0.020833343267440796 |
| eggroll_mean_final_eval_accuracy | 0.375 |
| eggroll_mean_final_eval_prompt_kl | 0.07928069680929184 |
| eggroll_mean_final_train_accuracy | 0.25 |
| eggroll_mean_initial_eval_accuracy | 0.3958333432674408 |
| eggroll_mean_initial_train_accuracy | 0.25 |
| eggroll_mean_max_eval_prompt_kl | 0.07928069680929184 |
| eggroll_mean_mean_fitness_std | 0.9060318917036057 |
| eggroll_mean_mean_weight_relative_update | 0.008577697077394617 |
| eggroll_mean_train_accuracy_change | 0 |
| eggroll_sigma | 0.01 |
| epochs | 4 |
| eval_batch_size | 32 |
| eval_samples | 96 |
| generations_per_prompt | 4 |
| max_new_tokens | 512 |
| mean_eval_accuracy_change_gap_decomposed_minus_eggroll | -0.041666656732559204 |
| mean_final_eval_accuracy_gap_decomposed_minus_eggroll | -0.041666656732559204 |
| mean_final_eval_prompt_kl_gap_decomposed_minus_eggroll | 0.26125362515449524 |
| num_decomposed_lr_magnitudes | 2 |
| num_ranks | 1 |
| num_seeds | 1 |
| num_target_modules | 7 |
| population | 32 |
| prompts_per_epoch | 8 |
| wall_clock_seconds | 963.4456016710028 |
Interpretation
Run 20260809-165410-qwen05_gsm8k_destructive_update_diagnostic_jean_zay-4b5ea4f supports the destructive-update suspicion. Both methods start from the same small-sample eval accuracy and lose accuracy after four updates, while prompt-logit KL from the initial model grows monotonically. Decomposed EGGROLL is worse in this setting: it loses more eval accuracy than plain EGGROLL and has much larger final KL drift, especially at lr_magnitude=0.35.
The boring explanation is not wrapper attachment, because the earlier protocol control showed zero no-update gap. Here the per-epoch records show a more direct problem: perturbed train exact-answer accuracy is zero in every epoch, so the ES update is not using exact-answer successes. The nonzero fitness variance comes from parseability shaping, meaning the optimizer is mostly learning from “produces a parseable number” rather than “gets the answer right.” That is enough to move the model distribution, but not enough to improve GSM8K correctness.
Next, do not run another broad LR sweep on this reward setup. The next diagnostic should remove or isolate parseability shaping and add a sign/zero-control comparison on the same fixed batch: zero LR, normal LR, and negative LR. If exact rewards remain all zero, the benchmark is too sparse for this early ES loop; use an easier reward curriculum or a denser objective before interpreting decomposed-vs-plain differences.