Service · AI and ML engineering

AI agents and evaluation harnesses

Agents that do multi-step work in your systems, with a test harness that proves they do it right.

Relevant to Service businesses, Growing businesses, Manufacturing, Education

The problem

You have seen a demo where an AI reads an email, checks the ERP and drafts a reply. Then it did the wrong thing on the third real example, and nobody could say how often that happens.

An agent is a system that takes several steps and uses your tools to finish a task. It is only as good as the harness that tests it: the set of real cases, the checks on each step, and the record of what it did.

What it costs today

Typical: a pilot built on a dozen hand-picked examples, launched, and quietly switched off after the first costly mistake. The money was spent and the trust is gone.

This is for you if
  • A task that takes a person several steps across two or more systems, many times a day: process a purchase order, reconcile a payment, prepare a quotation, triage a ticket.
  • You can describe what a correct result looks like, and you have past examples.
  • You accept a supervised phase where the agent proposes and a person approves.

Probably not for you if

  • A task where a wrong action cannot be undone and there is no place for a checkpoint. We design a human approval into it, or we decline.
  • A task whose steps never vary. That is automation, it is cheaper, and we will point you there.
What we build
  • An agent that completes the task end to end, using your systems through controlled connectors, with a human checkpoint wherever an action cannot be undone.
  • An evaluation harness: a few hundred real cases with known correct results, automatic graders for each step, and a regression run before every change.
  • A full trace of every run: what it read, what it decided, what it did, what it cost.
  • Guardrails on scope, spend and actions, and a kill switch your team controls.
How we build itIn this order
  1. 01

    Write the task as a checklist

    What the agent reads, what it decides, what it is allowed to do, and what a correct result is. The checklist becomes the grader.

  2. 02

    Build the test set before the agent

    Two to five hundred past cases with the right answer, drawn from your records. We run the first agent against it on day one and publish the score.

  3. 03

    Give it tools, not access

    Each action is a narrow function with limits: read this, update that field, never delete. Anything irreversible waits for a person.

  4. 04

    Iterate against the harness

    Every prompt, model or tool change is a regression run. The score goes up or the change does not ship. This is where most of the engineering goes.

  5. 05

    Supervised pilot, then widen

    The agent proposes, your team approves, and every disagreement becomes a new test case. Approval is removed one action at a time, as the harness earns it.

Data you already haveThis is usually enough to start
  • Past cases of the task, with how they were handled
  • The systems the task touches: ERP, CRM, ticketing, email, spreadsheets
  • The SOP, or the person who knows it
  • Any record of past errors and what they cost
In practice

Illustrative example

A quotation agent trusted on routine enquiries, because a test harness proved it first

Enquiries arrive by email and WhatsApp: a list of sections, grades and quantities, and a delivery location. Each one needs a stock check, the day's price list, freight and a reply. The sales team spends most of the day on enquiries that look almost identical.

Read the case study
Typical engagement
Duration2-week assessment, then 8 to 12 weeks to a supervised pilot
TeamML engineer, software engineer, your process owner
You receiveWorking agent on one task, harness with published scores, trace log, cost per task
PriceFixed price, quoted after the readiness assessment
What you own at the end
  • The agent, its connectors and its harness, in your repository and on your cloud account or server.
  • The test set and graders, so your team can evaluate a new model in an afternoon.
  • The trace log, the cost dashboard and the kill switch.

Everything runs in your environment and is handed over with documentation. Nothing stops working if we part ways.

Questions we are usually asked

Which models do you use?

Whichever scores best on your harness at a cost you accept, which is the point of having one. We have no commercial tie to any vendor, and the harness makes switching cheap.

How do we know it is safe to let it act?

It does not act until the harness shows a pass rate you have agreed, on your own cases, and irreversible actions always wait for a person. You see every run.

Is this not just automation?

If the steps are fixed, it is, and we build automation instead; it is cheaper and more reliable. An agent earns its place when the inputs vary and judgement is needed between steps.

What does a run cost?

It is measured and shown per task. Routing simple cases to small models, caching and short contexts usually keep it to a few rupees. The harness tells us when a cheaper model is good enough.

Can you evaluate an agent someone else built for us?

Yes. Building the harness around an existing agent is a common four to six week engagement, and it usually ends the argument about whether it works.

Often combined with

Further reading

Not sure if your data is ready?

Take the free 20-minute readiness check. You get a short scorecard and a plain answer about what, if anything, is worth doing next.

Start the readiness checkTalk to us