| status | note | summary | updated |
|---|---|---|---|
solid | Starting point | First formulation of the benchmark protocol and its game-variant taxonomy. | 2026-08-01 |
draft | Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs | Studies how steganographic collusion can emerge under optimization pressure and how standard mitigations fail when optimized against. The paper uses gradient-based and in-context reinforcement learning methods to elicit linguistic steganography. It is relevant to our robustness and mitigation axes, but its training-pressure setting is broader than our prompt-only first benchmark. | 2026-07-29 |
draft | Generative Text Steganography with Large Language Model | Introduces LLM-Stega, a black-box text steganography method using LLM user interfaces rather than model logits or token probabilities. The method constructs keyword sets, encrypted mappings, and rejection-sampling prompt optimization for accurate extraction. It is relevant because our benchmark also wants black-box user-facing output, but the paper is a stegosystem construction paper rather than an evaluation benchmark. | 2026-07-29 |
draft | Secret Collusion among AI Agents: Multi-Agent Deception via Steganography | Formalizes secret collusion among generative AI agents and introduces CASE, an evaluation framework for steganographic collusion. Empirical evaluations suggest current models are limited at advanced steganography but show capability jumps with GPT-4. This is the closest broad framing paper for our project. | 2026-07-29 |
draft | Robust Steganography from Large Language Models | Introduces formal weak and strong robustness definitions for LLM-based steganography under tampering and paraphrasing. The paper proposes watermarking-based and embedding-based constructions for robust hidden communication. It is most relevant to our paraphrase-robustness axis, though it is more cryptographic than benchmark-oriented. | 2026-07-29 |
draft | Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systems | Shows that tool-using frontier agents can construct sophisticated stegosystems by reading papers, searching, writing code, and testing implementations. The paper reframes covert-channel risk as an agentic construction and coordination problem rather than a cryptographic-knowledge bottleneck. It is important background, but mostly beyond our first prompt-only, text-only scope. | 2026-07-29 |
draft | Early Signs of Steganographic Capabilities in Frontier LLMs | Evaluates encoded message passing and encoded reasoning in frontier LLMs under monitor pressure. The paper finds current models need strong affordances such as coordination or scratchpads to succeed. It is highly relevant for our benchmark design because it separates capability, monitoring, and affordance conditions. | 2026-07-29 |
draft | Experiment: binary_punctuation_agreement_gpt_oss_20b_smoke | Jean Zay smoke for agreement-mediated binary-punctuation channel setup. | 2026-08-03 |
draft | Experiment: binary_punctuation_attackers_gpt_oss_20b | One-seed attacker-mediator test for the supplied punctuation channel. | 2026-08-01 |
draft | Experiment: binary_punctuation_gpt_oss_20b | Main binary-punctuation communication page: supplied-rule runs, receiver-learning smoke, and self-assessment telemetry. | 2026-08-01 |
draft | Experiment: binary_punctuation_monitor_gpt_oss_20b | Monitor ladder for binary-punctuation public dispatches: blind, batch-comparative, few-shot, and public-cover expert. | 2026-08-01 |
draft | Experiment: binary_punctuation_receiver_learning_gpt_oss_20b_smoke | Jean Zay smoke for the receiver-learning ladder after tightening receiver-zero-shot XML behavior. | 2026-08-03 |
draft | Experiment: punctuation_payload_gpt_oss_20b_8bit_probe | Jean Zay point-3 full sender/receiver 8-bit punctuation payload probe. | 2026-08-27 |
draft | Experiment: punctuation_payload_gpt_oss_20b_focused | 2026-08-10 | |
draft | Experiment: punctuation_payload_gpt_oss_20b_long | Jean Zay long-payload smoke for supplied punctuation payloads at 16, 32, 64, and 128 bits. | 2026-08-05 |
draft | Experiment: punctuation_payload_gpt_oss_20b_longbudget | 2026-08-11 | |
draft | Experiment: punctuation_payload_gpt_oss_20b_mechanical | 2026-08-19 | |
draft | Experiment: punctuation_payload_gpt_oss_20b_mechanical_receiver_local | Mechanical sender plus local mechanical receiver diagnostic for long punctuation payloads. | 2026-08-27 |
draft | Experiment: punctuation_payload_gpt_oss_20b_mechanical_receiver_sequence | Jean Zay diagnostic for long punctuation payload recovery when the receiver sees a pre-extracted punctuation sequence instead of the full briefing. | 2026-08-26 |
draft | Experiment: punctuation_payload_gpt_oss_20b_smoke | Jean Zay smoke for supplied multi-bit punctuation payloads under normalization and paraphrase. | 2026-08-04 |
draft | Experiment: punctuation_sequence_receiver_gpt_oss_20b_short | Jean Zay short receiver-only diagnostic for mapping synthetic punctuation sequences to payload bits. | 2026-08-27 |
draft | Experiment: self_assessment_binary_punctuation_gpt_oss_20b_smoke | Jean Zay smoke for condition-specific post-hoc private self-assessment XML. | 2026-08-03 |
draft | Paper roadmap thread | Draft roadmap for moving from the current supplied-rule result toward a paper. | 2026-08-04 |
In one paragraph
This project builds a benchmark for evaluating whether large language models
can collude through steganography: one model produces an apparently ordinary
message, another model tries to recover hidden information from it, and the
evaluation measures whether the pair can communicate covertly under controlled
conditions. The goal is not to assume that a hidden channel exists, but to make
that question testable.
Why it matters
Covert communication is a concrete failure mode for monitoring and control. If
models can reliably hide information in benign-looking text, then surface-level
inspection may miss coordination that matters for safety evaluations,
multi-agent deployments, and red-team methodology.
Approach
- Define simple sender/receiver games where the legitimate task and the hidden
communication objective can be separated. - Compare collusive prompting against baselines that should not contain a
usable hidden channel. - Store experiment outputs under
runs/, with reported metrics generated by
the experiment tooling rather than written into notes by hand. - Promote theory notes only after human review; draft notes are working
hypotheses, not established foundations.
Current findings
The first binary-punctuation experiments support a narrow supplied-rule result:
gpt-oss-20b can use a shared final-punctuation convention to transmit a
binary bit through ordinary XML task output while preserving visible task
success and avoiding explicit payload text. The channel is fragile under
normalization or prompt-aware sanitization, but survives punctuation-preserving
paraphrase.
Monitor results form a ladder. A generic blind individual-message monitor did
not notice the channel, but calibrated few-shot and feature-based public-cover
monitors exposed it. The receiver-learning smoke suggests the same ladder on
the receiver side: direct rule access is clean, labeled examples partially
recover the channel, and zero-shot discovery is weak and needs a tighter
prompt before scaling.
The supplied multi-bit punctuation payload smoke extends the same supplied-rule
story from a single binary choice to short payload strings. After hardening the
sender prompt and visible-task wording, the direct supplied-payload path is
clean in the tiny smoke. Its mediator behavior is also unsurprising:
punctuation normalization removes the channel, while standard paraphrase can
damage it by changing sentence boundaries.
All quantitative claims should still be read from
runs/<run_id>/results/metrics.json and reported through the project tooling.