IBM Research tops tau2-bench customer service benchmark by giving a GPT-4.1 agent learned memory
IBM Research has published results for ALTK-Evolve, a framework that improves AI customer service agents by appending automatically mined behavioral guidelines to the agent's prompt instead of retraining the model. The work was published on the Hugging Face blog on October 20, 2026.
The headline result: a basic ReAct agent running GPT-4.1, an older non-reasoning model, reached 67.5% Pass^4 on the retail domain of tau2-bench, Sierra's customer service agent benchmark, scoring above every retail entry in the benchmark's October 12, 2026 leaderboard snapshot.
Pass^4 is the benchmark's strict reliability metric: a task counts as solved only if the agent completes it correctly on all four independent attempts. IBM argues this matters because a support agent that resolves an issue three times out of four still fails a quarter of real customers.
The setup used tau2-bench's own stock ReAct scaffold with gpt-4.1 as the agent model and gpt-5.5 as the customer simulator, four trials per task. ALTK-Evolve analyzes the agent's training-task attempts and distills successes and failures into short rules, such as how to authenticate a customer or handle a return, which are then added to the system prompt unchanged for held-out test tasks.
Across both benchmark domains, mean accuracy rose from 78.1% to 82.5% on retail and from 57.5% to 65.0% on airline tasks, while Pass^4 climbed from 57.5% to 67.5% on retail and from 35.0% to 45.0% on airline. IBM reports the gains come from better-informed decisions rather than longer conversations, though the approach does increase token consumption.
For teams evaluating agent tooling, the result suggests learned prompt-level memory can lift an older model past frontier-model entries on structured support tasks without fine-tuning or human annotation. ALTK-Evolve is available through IBM's AgentToolkit on GitHub. The results are limited to tau2-bench's retail and airline customer service domains and do not measure performance on open-ended tasks.