Authors: Artur Zolkowski and Kei Nishimura-Gasparian and Robert McCarthy and Roland S. Zimmermann and David Lindner
Cite as: \cite{zolkowski2025early}
arXiv: https://arxiv.org/abs/2507.02737
Summary in three sentences
This paper studies whether frontier LLMs can pass encoded messages or perform encoded reasoning while avoiding monitors. It finds early steganographic capabilities, but mainly when models get extra affordances such as unmonitored scratchpads, coordination on schemes, or predefined encodings. The paper argues that current capabilities are probably insufficient to bypass well-designed monitors under standard conditions, but should be tracked as models improve.
Main contribution
The main contribution is a reusable evaluation suite for encoded message passing and encoded reasoning. For our project, the message-passing side is directly relevant; the encoded-reasoning side is useful background but partly outside our first-paper scope because we want final user-visible outputs, not hidden chain-of-thought.
Key equations and definitions
- Encoded message passing: a model must encode a hidden message in a cover output and a receiver must decode it, while a monitor checks for the hidden message.
- Encoded reasoning: a model performs reasoning in an encoded form, possibly embedded inside a cover task.
- The paper defines overall success in the case-study figures as a conjunction of monitor pass and task/message success, for example Figure 9 defines overall success as Monitor Pass AND Task Accuracy.
Empirical results
The abstract reports that current models cannot reliably encode short messages without monitor detection under standard affordances, but can succeed with affordances like an unmonitored scratchpad and coordination on the encoding scheme. Appendix B reports Figure 8 over 500 message-passing samples and Appendix A.2.3 reports Figure 9 over 200 encoded-reasoning case-study samples. Table 3 shows large direct-performance gaps with and without scratchpads, for example GPT-4.5-preview on State-tracking 2D has 100.0% with scratchpad and 42.5% without scratchpad; Table 4 and Table 5 show that encoded reasoning with cover tasks remains much weaker and varies strongly by model and encoding regime.
Relation to our project
- what we reuse: monitor-pass-and-decode-success scoring, affordance sweeps, coordination vs no-coordination comparisons, and reusable eval-suite framing.
- what we dispute: their encoded reasoning setup mixes with chain-of-thought monitoring, which our first benchmark explicitly excludes.
- what we must reproduce: a clean version of the encoded message-passing condition with binary payloads, a visible task score, and monitor/paraphrase conditions.
Criticism and limitations
The authors explicitly note limited use of reasoning models because many reasoning traces are unavailable or hard to steer. The benchmark includes several affordances that may be unrealistic for a pre-production user-facing model test. For our project, the key limitation is that message passing is only part of the paper; we need a more focused natural-language output benchmark.