Every agent demo looks the same: an email arrives, the system reads it, checks a few records and does the right thing. The demo is real. The question nobody asks is how often, and on which cases it does the wrong thing. The harness is the answer to that question, and we will not build an agent without one.
What a harness is
A harness is three things. A set of past cases, two to five hundred of them, each with the result a competent person produced. A grader for each step: did it read the right fields, did it pick the right action, did it stop where it should have asked. And a runner that puts a new version of the agent through all of it and prints one score, before anyone sees the change in production.
Why it comes first
The first agent we run against the harness scores badly. That is fine; it is the point. From then on every prompt change, every model swap, every new tool is a number going up or down, not an opinion. Engineers stop arguing about examples and start fixing the cases that fail.
The swap a harness catches
Take a purchase-order agent that has run for months with a person approving every action. A vendor releases a new model, cheaper and said to be better. Without a harness the swap gets made on a Friday afternoon because it is cheaper. With one, it is a regression run. If the score drops, and in our experience it drops on the awkward cases, orders with partial deliveries or a changed delivery address, the old model stays, the awkward cases become new tests, and the switch happens when the new model passes them.
What goes into the cases
- The ordinary ones, because most of the volume is ordinary and the agent must be boring there.
- Every case that went wrong by hand in the last year. The team remembers them; write them down.
- The ones where the right answer is to stop and ask a person. An agent that never asks is not safe, it is lucky.
- Cases that arrived during the pilot and were disputed. Each disagreement between the agent and the approver becomes a test case the same day.
What it costs
Building a first harness takes two to three weeks of the engineering time in an eight to twelve week project. It is the best-spent part. It is also the part that lets your own team evaluate a new model in an afternoon after we have gone, which is the point of handing it over.