ToolPrivacyBench: Benchmarking Purpose-Bound Privacy in Tool-Using LLM Agents

Authors: Shijing Hu and Liang Liu and Zhu Meng and Zhicheng Zhao
Cite as: \cite{hu2026toolprivacybench}

arXiv: https://arxiv.org/abs/2606.28061

Summary in three sentences

ToolPrivacyBench is a benchmark that audits purpose-bound privacy in multi-tool
LLM agents: it checks whether each private field (“atom”) is transmitted only to
the tools and downstream sinks that actually need it for their stated purpose.
Rather than checking the final answer, it executes agents against mock backends
and audits the full tool-call trajectory (arguments + backend audit logs)
against a per-case policy knowledge base. Across nine agents it finds that high
task-completion coexists with substantial over-disclosure, so successful tool
use does not imply privacy compliance.

Main contribution

• Formalizes a current-task privacy failure mode: private atoms necessary for
  one workflow step get needlessly routed to other tools, sinks, or free-text
  fields that do not require them (a need-to-know disclosure boundary).
• A knowledge-based auditing benchmark: 2,150 cases (1,150 fully synthetic
  privacy-sensitive business workflows + 1,000 adapted from existing multi-tool /
  function-calling benchmarks) across domains (healthcare, finance, insurance,
  tax, recruiting, education, IT ops, software security).
• Trajectory-level auditing via a policy knowledge base (used only post-hoc for
  audit, not given to the agent as a defense) + fine-grained metrics.

Key equations and definitions

• Private atom: a discrete piece of task-private information.
• Field-tool authorization matrix: specifies which field may be sent to which
  tool under which purpose (§3.4). Privacy is evaluated at the field-tool level,
  not as a global property of a field.
• Purpose-bound over-disclosure (§3.6): a private atom reaching a tool/sink
  for which it is not authorized under the current purpose.
• Metric families (§5): $\text{TaskSuccess}$ (completion, workflow coverage,
  required-fact delivery); $\text{FOR}$ = field opportunity-normalized
  over-disclosure rate (§5.2); severity-weighted leakage (§5.3); tool-level and
  leaky-tool-call rates (§5.4–5.5); free-text over-disclosure (§5.6);
  mid-workflow over-disclosure (§5.7); $\text{MT-POI}$ = multi-tool privacy
  over-disclosure index (§5.8); safety-maintained task completion (§5.9).

Empirical results

Every number here must name the section or figure it comes from.
On the synthetic private split, across nine agents (reported in §1, consistent
with abstract):

• $\text{TaskSuccess}$ ranges from 92.23 to 97.70 — most models complete
  multi-tool tasks.
• $\text{MT-POI}$ stays between 19.19 and 28.04 — completion does not imply
  appropriate disclosure.
• Aggregated $\text{FOR}$: 51.43 for tickets and 34.79 for handoffs
  (§1), identified as the most frequent leakage locations; free-text business
  fields also repeatedly act as over-disclosure channels.
  (Full per-model tables are in App. B; not transcribed here.)

Relation to our project

This is the closest published work to our research question (measuring the
“native” propensity of LLM agents to over-disclose private information while
using tools). Notably, they state they execute tasks through the OpenClaw
stack with mock backends (§1).

• what we reuse: the field-tool authorization framing and the idea of
  trajectory-level auditing (tool arguments + backend audit logs) rather than
  final-answer inspection; their metric vocabulary (FOR, MT-POI) as candidate
  metrics for quantifying intrusive-tool propensity.
• what we dispute / open questions for us: they audit over-disclosure of
  authorized fields to wrong sinks, whereas our question also covers choosing
  intrusive/illegal tools to succeed — related but not identical. Their policy
  KB is post-hoc audit only; our “RGPD-judge supervisor” idea would instead act
  at execution time, so we differentiate on the intervention side.
• what we must reproduce: whether the “task success ≠ privacy compliance”
  gap holds on Llama / GPT / Claude in our own scenarios; whether their
  synthetic-vs-public split matters for the measured propensity.

Criticism and limitations

• Heavily synthetic (1,150/2,150 fully synthetic workflows) — realism of the
  over-disclosure signal depends on how the policy KB was annotated (§4.4, §6.2).
• The policy KB is authored by the benchmark builders; “necessary vs.
  unnecessary” disclosure is a judgment call and may not generalize across legal
  regimes (RGPD vs. others).
• Post-execution audit only; it measures the failure but does not test
  execution-time defenses.
• Numbers here are from the intro/abstract; full model-level breakdown and the
  detector’s own reliability (App. D) should be checked before we cite specifics.