---
title: "Microsoft and Hugging Face launch ThinkingBox benchmark to grade AI agents on database outcomes"
slug: "microsoft-and-hugging-face-launch-thinkingbox-benchmark-to-grade-ai-ag"
date: "2026-10-05"
category: "AI & Automation"
author: "Marcus «Glitch» Reed"
correspondent: "Marcus «Glitch» Reed"
correspondentRole: "Open Source & Hacker Scout"
readingTimeMinutes: 1
tags: ["AI","News"]
originalSourceUrl: "https://huggingface.co/blog/microsoft/thinkingbox"
canonicalUrl: "https://aurav.ae/microsoft-and-hugging-face-launch-thinkingbox-benchmark-to-grade-ai-ag"
publisher: "Aura-V Intelligence (aurav.ae)"
---

# Microsoft and Hugging Face launch ThinkingBox benchmark to grade AI agents on database outcomes

> A new benchmark evaluates AI agents by the records they leave in back-end databases, offering B2B teams a clearer reliability signal.

## Key Takeaways (TL;DR)

- ThinkingBox grades agents on terminal state, not just tool calls, across 507 workflows run 20 times each.
- Over 121 k trials, 79 853 attempts failed executable checks, revealing a large gap between apparent success and actual data changes.
- Model rankings shift when consistency is measured; Claude Opus 5.5 leads overall at 67.16 % pass@1 but only 241 tasks achieve perfect 20-out-of-20 consistency.

Microsoft and Hugging Face have released ThinkingBox, a benchmark that grades AI agents on the database records they modify rather than the sentences they generate, a development that directly impacts B2B teams deploying autonomous agents [Microsoft and Hugging Face release ThinkingBox](https://huggingface.co/blog/microsoft/thinkingbox). The system runs agents through 507 stateful business workflows, each executed 20 independent times from a clean backend, and evaluates three metrics: pass@1 (single-attempt success), pass@20 (success at least once), and observed 20/20 (success on every run).

Model performance varies dramatically when consistency is considered. Open-weight model Kimi-K3 solves 93.89 % of tasks at least once yet records consistent success on just 13.41 % of them, highlighting the trade-off between breadth and reliability.

---

*Original Think-Tank Source: [The Agent Said It Was Done. The Database Disagreed.](https://huggingface.co/blog/microsoft/thinkingbox)*