-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Source
- Person key:
ysymyth - Source kind:
paper - Canonical URL: https://arxiv.org/abs/2406.12045
- License:
NOASSERTION - Public handling:
public-metadata-summary-hash-link-only - Semantic hash:
4e98d407b2077213875f40fb43c3a4bb77a7c37be710d3119effff1d96b46b90 - First seen: 2026-05-16
- Last changed: 2026-05-16
- Identity guard: Do not confuse with yao-shunyu-alfred, the physics-to-AI researcher at alfredyao.github.io.
Classification
- Category: Language agents / agent architectures
- Topic hub: shunyu-yao-public-corpora
- Project taxonomy: shunyu-yao-project-taxonomy
- Paper map: shunyu-yao-paper-map
Summary
Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose -bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with t…
What This Teaches
How language models become agents through reasoning, acting, memory, tools, and interface design.
Related
Public Handling Notes
- EXTRACTED: Metadata and links are from public sources.
- INFERRED: Unclear-license full text, PDFs, and source code should not be mirrored into public wiki pages.
- AMBIGUOUS: Items discovered by keyword search are still separated by source identity and category heuristics.