The best AI agent in the world just scored 48.6%

A number came out this week that should worry anyone who has handed real work to AI, and it’s the best news I’ve had all month. Let me explain both halves.

What was the test, in plain words?

A new public test (called AutomationBench, built by the automation company Zapier with the research firm Artificial Analysis) gave AI agents 657 real office tasks across 40 everyday business apps: invoices, spreadsheets, customer records, calendars. Real work, not puzzles.

The best AI in the world finished 48.6% of the tasks. Less than half. Money-related tasks were the hardest. And every single AI tested broke rules it was explicitly told to follow, on average about once per task.

Shouldn’t that scare someone running a business on AI?

I run everything on 13 AI agents, so you’d think that number would keep me up at night. It’s the opposite. It confirms the design.

I never bet on one AI being right. Every finished job is double-checked by a second AI from a different company: the one who does the work never approves it. Anything that touches money, sending, or publishing stops and waits for my yes. And every AI here has standing permission to refuse a task when the instructions don’t add up. A refusal is a good result, not a failure.

What is the test actually measuring?

Not whether AI is useful. It’s measuring what happens when you trust one AI, alone and unsupervised. Half-right and rule-breaking is that answer, and it matches what I see in my own logs every week.

The winning move was never a smarter AI. It’s what you build around an imperfect one: a second checker, a human yes on anything that matters, and a clear path to refuse.

48.6% isn’t a verdict on AI. It’s the strongest argument yet for architecture over horsepower.

I build AI agents that run entire departments, and I document the whole build here, what works and what breaks. If you want the same agents installed in your own business, here’s how we’d work together.