How to test AI before it touches your operations

A demo that impresses a steering group is not evidence that an AI system is ready for your invoice queue. Live operations will send it the badly scanned page, the contract with two effective dates and the supplier who changed its bank details last month. Testing is how you find out what happens then, before a customer or an auditor does.

What does testing an AI system actually mean?

Conventional software testing asks a yes or no question: given this input, did the system return exactly this output? AI systems do not behave that way. The same input can produce different wording on different days, and "correct" is often a judgment rather than a string match. So AI testing measures behavior across many cases instead of checking one.

The usual term is an evaluation, often shortened to an eval: a set of test inputs, the expected result for each, and a way of scoring what the system produced. OpenAI describes evals as tests that check model outputs against the style and content criteria you specify, and says they matter most when you are upgrading or trying new models. Three scoring methods exist, and most programs use all three.

• Rule-based checks. Structured comparisons: did it pull the right invoice number, is the total a number, did it choose one of five allowed categories. Cheap, fast and unarguable.
• Model-graded checks. A second model scores the output against a written rubric, which suits summaries and drafted replies where wording varies but substance should not. Amazon Bedrock offers evaluation jobs that use a judge model, which scores each response and explains the score, and Microsoft Foundry ships built-in evaluators for groundedness and relevance.
• Human review. People who know the work rate a sample. Slowest, most trustworthy, and the only way to settle cases where two experts disagree. Bedrock also supports evaluation jobs staffed by your own subject matter experts.

How do you build a test set from real past cases?

The test set is the most valuable artifact in the project, and it has to come from your own history rather than from examples a vendor wrote. Examples written by the builder describe the system that was built.

1. Pull a real period of work. A month of invoices, a quarter of tickets, a year of engagement letters, long enough to contain a seasonal oddity.
2. Put the exceptions in on purpose. A test set of clean cases proves nothing. Ask the team for the ones that caused arguments.
3. Write down the right answer for each case. This is the ground truth, and it is the part people skip. Bedrock's evaluations for retrieval-based systems require a dataset containing the expected retrieved text and response, so the score means something.
4. Redact what you must, keep what matters. Mask sensitive fields, but do not tidy the documents. The smudges are the test.
5. Freeze it. Microsoft's model migration guidance is blunt: keep inputs, ground truths and success criteria fixed for the whole exercise, because once any of them change you can no longer compare one run with another.
6. Hold some back. Keep a portion the people tuning the system never see, so you can tell tuning from overfitting.

Fifty to a hundred cases per task type is enough to start and small enough to actually label. You add to it forever afterward, because every production failure becomes a new test case.

Illustrative example: Consider a freight forwarder that wants AI to read delivery notes and match them to shipments. The operations manager exports 200 documents from the previous quarter: 120 routine, 80 that generated a query at the time. For each, a coordinator records the correct shipment reference, the correct piece count and whether the document should have been escalated. That labeling takes two people about a day and a half, and it becomes the acceptance test, the regression suite and the evidence in a later conversation with the carrier about which documents are unreadable at source.

What acceptance criteria should you agree with the business owner?

Acceptance criteria decide whether the system goes live, so they belong to the person accountable for the work, not to whoever is building it. Agree them in writing before development starts, in the language of the process. Four questions settle most of it:

• What counts as correct? For extraction, field by field. For a drafted reply, what must be present and what must never be.
• What is the target, measured against today? A target of 95 percent means nothing until you know whether people currently manage 88 percent or 99 percent.
• Which errors are unacceptable at any rate? Paying the wrong bank account, quoting a price that does not exist, telling a customer their warranty has expired. These get their own rule, not a percentage.
• What happens when the system is unsure? Every criterion needs a defined route for low confidence, missing data or a failed integration, and a named queue to land in.

If a supplier cannot hold a conversation at this level of detail, they are not ready to build. What happens in an AI readiness assessment covers how to reach these numbers before a build is scoped.

How do you measure accuracy and the failures that matter?

"Accuracy" is too blunt a word to decide with. A 5 percent error rate made of missing fields is a nuisance. A 5 percent error rate made of invented figures is a serious problem. Count the failure types separately.

Report the mix, not one number, then decide with the business owner which categories review can absorb and which must be engineered out before launch. The NIST AI Risk Management Framework, released in January 2023 and extended with a generative AI profile in July 2024, gives you governance vocabulary your risk team will recognize.

Shadow mode or a pilot: how should the first live weeks run?

Two stages sit between a passing test set and full operation, and they answer different questions.

Shadow mode runs the system on real live inputs while its output goes nowhere. Nothing is written to a system of record, and you compare what it would have done with what your team actually did. Azure Machine Learning implements this as mirrored traffic, also called shadow testing: a percentage of live requests is copied to the new deployment so you can gather metrics and logging without affecting clients, capped at 50 percent of traffic, while every client response still comes from the existing path. Microsoft's migration guidance recommends the same approach for regulated flows that cannot expose a new model to customer traffic at all.

A pilot puts the output in front of a small, named group who can act on it, with a clear route back. The value here is behavioral: do people trust it, do they use the override, do they understand its choices.

Set an exit condition for each stage before you enter it: the test set is passed when targets hold on the held-back cases, shadow mode when agreement with the team is stable over a defined period, and the pilot when reviewers stop correcting the routine cases. Keep the previous way of working available throughout. Microsoft calls this preserving a rollback corridor, and notes that a rollback forced by a deadline rather than by evidence is a sign the work started too late.

How much human review do you need, and for how long?

Start by reviewing everything, then earn your way down.

1. Review 100 percent of outputs for a defined opening period, or a defined number of items.
2. Move to a stratified sample: full review on high-value, low-confidence and exception cases, plus a random slice of the routine ones.
3. Record every reviewer correction. These are the most valuable new test cases you will get, and they are free.
4. Watch for reviewer disagreement. When two experienced people mark the same output differently, the problem is usually the criteria rather than the model.
5. Never remove human approval from consequential actions. Payments, contractual commitments and anything affecting a person's employment or credit keep a named approver.

How do you keep it working when prompts, models and processes change?

Regression testing

Treat anything that changes behavior as a release: prompt text, model version, retrieval settings, tool definitions and guardrail configuration. A one-line prompt edit can change outputs more than a code change does.

The discipline is simple. Run the frozen test set to get a baseline, make the change, run it again and compare. Microsoft's migration guidance adds a refinement: when you move to a new model, replay the existing configuration unchanged first, so you can separate what the model changed from what you changed. Score quality, latency and cost together, because a model that is more accurate and three times slower can still fail the business test.

Model changes are scheduled events rather than surprises. On Microsoft Foundry, generally available models carry a retirement date set 18 months from launch (12 months for models from Anthropic, DeepSeek, Fireworks and Mistral AI), with at least 60 days notice and no extensions. Standard deployment types can be auto-upgraded at retirement unless you configure otherwise, so a model change can reach production outside your release process. On Amazon Bedrock, models move through Active, Legacy and end-of-life states with a Legacy notice period of either 6 months or 45 days, and AWS states that migration will not happen automatically. Record these dates with an owner.

Monitoring after launch

Three things need watching, and they are usually owned by different people.

• Operational health. Errors, latency, throttling, token usage and cost.
• Quality. Sampled scoring of live traffic plus scheduled runs of the test set to detect drift. Foundry's observability documentation describes both, with alerts when thresholds are breached.
• The business measure. The number the sponsor agreed at the start: handling time, cycle time, exception volume, rework. If that has not moved after the ramp, the project has not worked whatever the evaluation scores say.

Add a tested off switch: a documented way to disable the AI path and return to manual handling without deploying code. More on that setup in running AI in production on AWS and Azure.

What are the red flags when a vendor has no testing plan?

• The demo is the evidence. A curated demo shows the system can succeed once, on data somebody picked.
• No test set from your data. If nobody has asked for real past cases, nobody plans to measure against your reality.
• Accuracy quoted before scoping. A percentage offered before anyone has seen a document is a sales figure.
• No named owner for acceptance criteria. If the business owner has not agreed what correct means, "done" gets decided by whoever is holding the invoice.
• No plan for model changes. Ask what happens when the underlying model is retired. A vague answer means a maintenance problem you will inherit.
• No route for uncertain cases. A system with no exception queue is a system that will guess.

For the wider set of questions to put to any supplier, see how to choose an AI implementation partner.

How Kastling tests AI before it reaches your operations

Kastling builds and tests against real business scenarios rather than sample data, as part of the AI Integration & Automation service. Engagements start with a free discovery call, and for AI and operations work a separately scoped paid audit usually comes next, which is where the workflows, systems and data behind a test set are identified.

In delivery, we agree acceptance criteria with the business owner before building, assemble test cases from the client's own history and involve the people who do the work in reviewing outputs. We keep a named person in control of consequential decisions and agree up front how success will be measured. Monitoring and ongoing improvements are scoped explicitly rather than assumed, under a separate agreement where they apply.