Design
Which solid notes this experiment tests, and how.
Results
20260819-133615-qwen3_4b_gsm8k_plain_eggroll_repro_jean_zay-5329581
commit: 5329581
| metric | value |
|---|---|
| completed | 0 |
| eggroll_lr | 1 |
| eggroll_sigma | 0.001 |
| epochs | 100 |
| eval_batch_size | 5 |
| eval_generations_per_prompt | 5 |
| eval_prompts | 268 |
| eval_samples | 1340 |
| eval_temperature | 1 |
| eval_top_k | 0 |
| generations_per_prompt | 6 |
| latest_epoch_epoch | 35 |
| latest_epoch_fitness_std | 0.720064640045166 |
| latest_epoch_rank | 1 |
| latest_epoch_raw_score_mean | 0.2857142984867096 |
| latest_epoch_raw_score_std | 0.45175397396087646 |
| latest_epoch_seed | 0 |
| latest_epoch_train_accuracy | 0.2857142984867096 |
| latest_epoch_train_parseable_rate | 0.9920635223388672 |
| latest_epoch_used_parseability_shaping | 0 |
| latest_validation_accuracy | 0.20820896327495575 |
| latest_validation_epoch | 30 |
| latest_validation_num_correct | 279 |
| latest_validation_num_parseable | 1339 |
| latest_validation_parseable_rate | 0.9992537498474121 |
| latest_validation_rank | 1 |
| latest_validation_seed | 0 |
| max_new_tokens | 1024 |
| noise_reuse | 4 |
| num_completed_runs | 0 |
| num_epoch_records | 35 |
| num_ranks | 1 |
| num_seeds | 1 |
| num_target_module_patterns | 8 |
| num_validation_records | 4 |
| parseability_shaping | 0 |
| population | 126 |
| prompts_per_epoch | 21 |
| temperature | 1 |
| top_k | 0 |
| train_batch_size | 6 |
| validate_every | 10 |
| validation_epoch_0_accuracy | 0.32611939311027527 |
| validation_epoch_0_parseable_rate | 1 |
| validation_epoch_10_accuracy | 0.7149254083633423 |
| validation_epoch_10_parseable_rate | 1 |
| validation_epoch_20_accuracy | 0.49850746989250183 |
| validation_epoch_20_parseable_rate | 1 |
| validation_epoch_30_accuracy | 0.20820896327495575 |
| validation_epoch_30_parseable_rate | 0.9992537498474121 |
| wall_clock_seconds | 71261.713226076 |
20260820-183000-qwen3_4b_gsm8k_plain_eggroll_repro_jean_zay-00ed9dd
commit: 00ed9dd
| metric | value |
|---|---|
| center_eval_do_sample | 0 |
| center_eval_epoch_0_eval_accuracy | 0.11567164212465286 |
| center_eval_epoch_0_eval_parseable_rate | 1 |
| center_eval_epoch_0_train_accuracy | 0.0476190485060215 |
| center_eval_epoch_0_train_parseable_rate | 1 |
| center_eval_epoch_10_eval_accuracy | 0.7388059496879578 |
| center_eval_epoch_10_eval_parseable_rate | 1 |
| center_eval_epoch_10_train_accuracy | 0.8571428656578064 |
| center_eval_epoch_10_train_parseable_rate | 1 |
| center_eval_epoch_20_eval_accuracy | 0.5074626803398132 |
| center_eval_epoch_20_eval_parseable_rate | 1 |
| center_eval_epoch_20_train_accuracy | 0.7142857313156128 |
| center_eval_epoch_20_train_parseable_rate | 1 |
| center_eval_prompts | 268 |
| center_eval_temperature | 1 |
| center_eval_top_k | 0 |
| center_train_prompts | 21 |
| completed | 0 |
| eggroll_lr | 1 |
| eggroll_sigma | 0.001 |
| epochs | 100 |
| eval_batch_size | 5 |
| eval_generations_per_prompt | 5 |
| eval_prompts | 268 |
| eval_samples | 1340 |
| eval_temperature | 1 |
| eval_top_k | 0 |
| generations_per_prompt | 6 |
| latest_center_eval_epoch | 20 |
| latest_center_eval_eval_accuracy | 0.5074626803398132 |
| latest_center_eval_eval_num_correct | 136 |
| latest_center_eval_eval_num_parseable | 268 |
| latest_center_eval_eval_parseable_rate | 1 |
| latest_center_eval_rank | 1 |
| latest_center_eval_seed | 0 |
| latest_center_eval_train_accuracy | 0.7142857313156128 |
| latest_center_eval_train_num_correct | 15 |
| latest_center_eval_train_num_parseable | 21 |
| latest_center_eval_train_parseable_rate | 1 |
| latest_epoch_epoch | 29 |
| latest_epoch_fitness_std | 0.7261000871658325 |
| latest_epoch_rank | 1 |
| latest_epoch_raw_score_mean | 0.3809524178504944 |
| latest_epoch_raw_score_std | 0.48562091588974 |
| latest_epoch_seed | 0 |
| latest_epoch_train_accuracy | 0.3809524178504944 |
| latest_epoch_train_parseable_rate | 1 |
| latest_epoch_used_parseability_shaping | 0 |
| latest_validation_accuracy | 0.49850746989250183 |
| latest_validation_epoch | 20 |
| latest_validation_num_correct | 668 |
| latest_validation_num_parseable | 1340 |
| latest_validation_parseable_rate | 1 |
| latest_validation_rank | 1 |
| latest_validation_seed | 0 |
| max_new_tokens | 1024 |
| noise_reuse | 4 |
| num_center_eval_records | 3 |
| num_completed_runs | 0 |
| num_epoch_records | 29 |
| num_ranks | 1 |
| num_seeds | 1 |
| num_target_module_patterns | 8 |
| num_validation_records | 3 |
| parseability_shaping | 0 |
| population | 126 |
| prompts_per_epoch | 21 |
| temperature | 1 |
| top_k | 0 |
| train_batch_size | 6 |
| validate_every | 10 |
| validation_epoch_0_accuracy | 0.32611939311027527 |
| validation_epoch_0_parseable_rate | 1 |
| validation_epoch_10_accuracy | 0.7149254083633423 |
| validation_epoch_10_parseable_rate | 1 |
| validation_epoch_20_accuracy | 0.49850746989250183 |
| validation_epoch_20_parseable_rate | 1 |
| wall_clock_seconds | 60604.74062844599 |
Interpretation
Both Qwen3 plain EGGROLL reruns timed out before completion, but the instrumentation recovered partial metrics. The validation trajectory is not a clean improvement: it improves early, then degrades by the latest recorded validation while parseability stays essentially saturated.
The center-model diagnostic confirms that the early gain is visible on the updated center model, not only in noisy population rollouts, but it also decays before timeout. Treat these runs as evidence that the streaming update and diagnostics work, and as a warning that this GSM8K exact-answer setup is unstable under the current hyperparameters, not as a successful reward-learning result.