ToolPrivacyBench: Benchmarking Purpose-Bound Privacy in Tool-Using LLM Agents

Authors: Shijing Hu and Liang Liu and Zhu Meng and Zhicheng Zhao
Cite as: \cite{hu2026toolprivacybench}

arXiv: https://arxiv.org/abs/2606.28061

Summary in three sentences

ToolPrivacyBench evaluates whether tool-using LLM agents respect purpose-bound privacy while executing multi-step workflows, instead of checking only task success or final-answer leakage. Each case represents private atoms, tool purposes, sink types, and field-tool authorization rules in a policy knowledge base, then audits executed tool arguments and mock-backend logs after the agent runs. The paper’s main claim is that high tool-use success can coexist with systematic over-disclosure into tickets, handoffs, free-text messages, notes, and summaries.

Main contribution

The paper formalizes current-task over-disclosure as a trajectory-level information-flow problem: a private atom may be authorized for one tool but forbidden for another depending on purpose and sink. It contributes ToolPrivacyBench, a benchmark with executable multi-tool workflows, mock business backends, backend audit logs, authorization annotations, and metrics that separate task utility from privacy leakage.

The benchmark has two splits: the Need-to-Know synthetic private split and a public-derived split adapted from existing tool-use benchmarks. The distinctive contribution is the field-tool authorization matrix and policy knowledge base, which make privacy evaluation local to the current workflow rather than treating sensitivity as a global binary label.

Key equations and definitions

Multi-tool workflow, from Section 3.1:

$W = (x, T, P, A)$

where $x$ is the user request, $T$ the tool set, $P$ the current-task private atoms, and $A$ the field-tool authorization matrix.

Tool representation, from Section 3.1:

$t_{i} = (nam e_{i}, p u r p os e_{i}, sc h e m a_{i}, s in k_{i})$

Private atom representation, from Section 3.2:

$p_{j} = (t y p e_{j}, v a l u e_{j}, a l ia se s_{j}, se v er i t y_{j})$

Field-tool authorization, from Section 3.4: $A_{j, i} = 1$ means private atom $p_{j}$ is authorized for tool $t_{i}$ in the current workflow; $A_{j, i} = 0$ means disclosure to that tool is forbidden.

TaskSuccess, from Section 5.1:

$T a s k S u ccess = (S_{t a s k} \cdot S_{w or k f l o w} \cdot S_{f a c t})^{1/3}$

Field Opportunity-Normalized Over-Disclosure Rate, from Section 5.2:

$F O R = \frac{\sum _{k = 1}^{K} \sum _{p_{j} \in P} D ( p _{j} , a _{k} ) I [ A _{j, i (k)} = 0 ]}{\sum _{k = 1}^{K} \sum _{p_{j} \in P} I [ A _{j, i (k)} = 0 ]}$

Multi-Tool Privacy Over-Disclosure Index, from Section 5.8:

$M T - P O I = 100 \cdot (λ_{1} F O R + λ_{2} S W L R + λ_{3} T oo l F O R + λ_{4} F r ee T e x tF O R + λ_{5} M i d F O R)$

Safety-Maintained Task Completion, from Section 5.9:

$S M T C = 100 \cdot T a s k S u ccess \cdot (1 - \frac{M T - P O I}{100})$

Empirical results

Every number here must name the section or figure it comes from.
If you cannot locate it in the paper, do not write it.

Dataset scale: Table 5 reports $1, 150$ Need-to-Know synthetic private cases and $1, 000$ public-derived cases, for $2, 150$ total cases. The same table reports $20, 817$ private-atom instances and $219, 986$ field-tool authorization pairs across both splits.
Annotation reliability: Table 6 reports $93.1%$ raw agreement overall and Cohen’s $κ = 0.86$ before adjudication on the sampled re-annotation set.
Overall model behavior: Section 7.1 reports that public-derived TaskSuccess ranges from $76.30$ to $94.72$ and MT-POI from $15.81$ to $22.56$ ; on the Need-to-Know synthetic private split, TaskSuccess ranges from $92.23$ to $97.70$ , while MT-POI ranges from $19.19$ to $28.04$ and FreeTextFOR from $27.16$ to $46.76$ .
Privacy-utility mismatch: Section 7.1 reports that Qwen3.6-plus, Kimi K2.5, and DeepSeek V4 Flash reach TaskSuccess of $97.70$ , $97.70$ , and $97.20$ on the Need-to-Know split, but all have MT-POI above $27$ ; Gemini 3.5 Flash has lower TaskSuccess at $92.45$ but the lowest MT-POI at $19.19$ .
Sink-level leakage: Section 7.3 and Figure 7 report tickets as the highest-risk sink, with aggregated FOR $51.43$ , SWLR $48.42$ , and FreeTextFOR $50.79$ .
Free-text leakage: Section 7.4 and Figure 7 report FTSlotRate above $80%$ for message, description, and work_notes; description has the highest FreeTextFOR at $45.82$ .
Path-level leakage: Section 7.5 and Table 9 report that the record -> ticket transition accounts for $37.27%$ of first-leak events and has a RiskyTransitionRate of $96.89$ .

Relation to our project

what we reuse: the purpose-bound framing, the distinction between task success and disclosure compliance, and the idea of auditing executed tool trajectories rather than final answers. The field-tool authorization matrix is a strong starting point for our own benchmark design.
what we dispute: the project should not inherit the paper’s policy labels uncritically. For our GDPR-oriented objective, “necessary for the tool purpose” must be connected to legal basis, data minimization, recipient role, retention, and possibly explicit user consent, not only workflow functionality.
what we must reproduce: the privacy-utility separation, especially a TaskSuccess-like measure paired with leakage metrics. We should reproduce the claim that models can complete tasks while over-disclosing, using our target models and a smaller set of French/GDPR-relevant workflows before scaling.

Criticism and limitations

The benchmark is a controlled audit, not evidence about real production incident rates: Section 9 states that the workflows use fabricated or test values, mock backends, and policy annotations based on stated tool purposes. The paper also depends heavily on the quality of authorization labels; Table 6 gives sampled reliability, but the full benchmark is not independently re-annotated.

The legal notion of privacy is thinner than what this project needs. The benchmark operationalizes need-to-know disclosure, which aligns with data minimization, but it does not fully model GDPR concepts such as lawful basis, purpose limitation across controllers/processors, retention, data subject rights, or cross-border transfer.

The evaluated model list includes future or not-yet-standard model names, so the empirical ranking should be treated cautiously until the benchmark artifacts and execution harness are available. The reusable part for us is less the exact leaderboard than the measurement design: policy-grounded trajectories, backend logs, and diagnostics by sink/free-text/path.

Quartz 5

Explorer