Authors: Xingwen Zhang and Jeff Clune and Kenneth O. Stanley
Cite as: \cite{zhang2017on}
arXiv: https://arxiv.org/abs/1712.06564
Summary in three sentences
This paper studies how the OpenAI evolution strategy relates to stochastic gradient descent. It uses controlled MNIST experiments to compare ES-derived update directions with SGD gradients and to build an SGD-based proxy for ES population-size effects. For this project, it is useful background for interpreting whether ES is acting like noisy gradient estimation or like a genuinely different optimisation procedure.
Main contribution
The main contribution is an empirical analysis of the relationship between OpenAI ES and SGD in a supervised setting where gradient comparisons are tractable.
Key equations and definitions
The relevant object is the ES gradient estimator formed from reward-weighted random perturbations. The project should revisit this paper before writing theory about when ES approximates gradient descent and when the approximation breaks.
Empirical results
Every number here must name the section or figure it comes from.
If you cannot locate it in the paper, do not write it.
The arXiv abstract reports MNIST-based experiments on ES/SGD gradient correlation and ES population-size behaviour. Exact reported accuracies or correlations should be copied only after reading the corresponding figures or tables.
Relation to our project
- what we reuse: the ES-versus-SGD comparison frame and gradient-correlation diagnostics.
- what we dispute: the supervised MNIST setting is too clean to stand in for long-context agent rewards.
- what we must reproduce: a small diagnostic showing whether the PyTorch ES implementation produces sensible update directions on a controlled objective.
Criticism and limitations
The setting is deliberately simple and does not include LLM adapters, delayed rewards, or long-context trajectories. Its value here is diagnostic and theoretical, not as evidence that ES will work for agent post-training.