Key takeaways
- Pilots usually stall for operational reasons visible from day one: no named owner, no baseline, a demo on clean data and no plan for exceptions.
- A pilot built to become production puts integration, a real test set and an approval design in scope from day one, which avoids paying for a second project later.
- Decide whether to scale, fix or stop against criteria agreed before the results come in, not after.
- Monitoring, access, support and cost per item all change between a pilot and production, and none of them are optional extras to add later.
- Naming who owns the system after launch, before the pilot starts, is one of the cheapest decisions that most affects whether it ever reaches production.
A pilot that impresses in a demo and a workflow that runs in production are different things, and the gap between them is where many AI initiatives quietly die. Some teams call this early stage a proof of concept, a limited test built to show an idea works before anyone commits to building it properly. The model worked. The steering group nodded. Then nothing changed about how the invoice, the ticket or the application actually moves through the business, because nobody built the parts a demo does not need: an owner, an integration, an approval design and a plan for what happens when the input is messy. This guide, written for operations and IT leaders at companies of roughly 20 to 200 people, covers why that gap opens, how to design a pilot that can close it from day one, and how to decide, with evidence rather than optimism, whether to scale, fix or stop.
How often do AI pilots fail to reach production?
No single survey counts how many AI pilots fail, but the available figures agree that using AI is common and running it at scale is not. Each row below measures something different, so read them as a pattern rather than a failure rate.
| Source | What it measured | Figure | Year |
|---|---|---|---|
| McKinsey, 1,993 respondents | Organizations reporting regular AI use in at least one business function | 88%, up from 78% a year earlier | 2025 |
| McKinsey, same survey | Respondents saying AI is fully scaled across their organization | 7% | 2025 |
| Stanford HAI AI Index | Surveyed organizations using AI, and AI agent deployment by function | 88%; single digits across nearly all functions | 2025 |
| Gartner, forecast | Agentic AI projects canceled | Over 40% by the end of 2027 | June 2025 |
Gartner attributes its cancellation forecast to escalating costs, unclear business value or inadequate risk controls. That is a forecast, not a count of completed failures, but it lines up with the reasons in the next section: most agentic AI work today is still an early experiment, not a production system with a named owner and a plan. Adoption is no longer the hard part. Turning a working pilot into part of how the business runs is.
Why do AI pilots stall before they reach production?
AI pilots usually stall for operational reasons that were visible on day one, not because the model failed. Eight patterns recur.
- No named business owner. A pilot run by an IT or innovation team, with no one from the process itself accountable for the result, has no one to decide what happens next.
- No baseline. Without a measurement of how the work is done today, cycle time, error rate, hours spent, there is nothing to compare the pilot against, so the result becomes a matter of opinion.
- A demo on clean data. Ten tidy documents prove the model can succeed once, on data somebody picked. Live work sends the badly scanned page and the contract with two effective dates.
- No integration into the systems where the work happens. A pilot that runs in a separate tool, with someone copying results back into the ERP or CRM by hand, is not automation. It is a manual step with an AI assistant attached.
- No plan for exceptions and approvals. A pilot that only shows the happy path has not answered the question that decides whether it is safe to run unattended: what happens when the system is unsure.
- Security review left to the end. A pilot that reaches the security or compliance team only when someone asks about production access restarts the clock.
- No one to run it after launch. A pilot built by a vendor or a consultant, with no internal owner named for monitoring and support, has no path into daily operation, whatever the results.
- Costs unknown at scale. A pilot processing fifty items a day says little about what a thousand a day will cost in model usage and review time, and finance will ask before agreeing to scale.
How do you design a pilot that can become production from day one?
A pilot designed to prove a point and a pilot designed to become production are different projects, and the second puts more work in scope at the start so there is no second project later. Seven decisions separate them.
- A real workflow, not a representative one. Pick the actual process a team runs today, with its actual volume and its actual exceptions, rather than a simplified version built to showcase the model.
- Real data from the start. Pull actual documents, tickets or requests from your own systems, including the awkward ones, rather than a curated sample.
- A success measure agreed before the pilot starts. State the target in the same units as your baseline, agreed with the business owner rather than the technical team.
- Integration in scope from day one. Even a lightweight connection to the ERP, CRM or ticketing system used every day tells you more than a standalone demo ever will, and building it later is a second project, not a formality.
- A test set built from real cases. The same fifty to a hundred documented examples used to test the system before launch double as the evidence for the scale decision. How to test AI before it touches your operations covers how to build one.
- The approval design, not only the workflow design. Decide which actions the system can take on its own, which need a named person to approve, and which stay with people entirely, before the pilot runs rather than after something goes wrong. The human approval planner works through this for one task at a time and returns a control pattern your team can set up.
- An owner for day two. Name who runs, monitors and improves the system after the pilot ends, before it starts. A pilot with no owner for what comes after is a demo with a longer runtime.
How do you decide whether to scale, fix or stop?
The end of a pilot is a decision point, not an automatic green light, and it works best against criteria agreed before anyone sees the results.
| Decision | When it applies | What has to be true |
|---|---|---|
| Scale | The pilot met its agreed target against baseline, exceptions route to a named owner, and cost per item holds up at expected volume | Integration, monitoring and support are scoped for daily operation, not only the pilot period |
| Fix | The pilot missed its target in a specific, explainable way, such as one document type or one exception category | The gap can be named, and closing it does not require redesigning the whole workflow |
| Stop | The pilot cannot reach an acceptable error rate on the cases that matter, the data needed does not exist in usable form, or no owner will take responsibility for it in production | Stopping now costs less than finding this out after a wider rollout |
Illustrative example: Consider a regional telecommunications provider piloting AI to draft replies to service activation requests. The pilot runs for six weeks against real tickets, with a named support lead reviewing every drafted reply before it is sent. At the end of the pilot, drafts need editing on a similar share of routine requests as the baseline the team recorded before starting, but replies to one specific request type, an address change bundled with a plan upgrade, are wrong often enough that the team routes that category to a fix list rather than calling the whole pilot a success or a failure.
Stopping is a legitimate outcome, not a failure to report quietly. A pilot that shows a workflow is not ready for AI, and explains why, has still done its job.
What is an AI pilot, and what changes when it goes to production?
An AI pilot is a time-limited trial of an AI system on a real workflow, run to decide whether to scale it, while production is the same system running daily with an owner, monitoring and a known cost per item. A pilot and its production version usually run on the same model, but almost everything around the model changes.
| Area | In a pilot | In production |
|---|---|---|
| Owner | The team that built it | A named business owner and a named support owner |
| Monitoring | Watched by the people who built it | Automatic alerts when quality or cost drifts, with no one needing to remember to look |
| Access | Often broader than it should be, because speed mattered more than governance | Named service accounts, least privilege and a record of who can see what |
| Support | Builders fix problems as they appear | Someone owns what happens when the system errors, a source system changes its layout, or the underlying model is retired or upgraded |
| Cost | Measured at pilot volume, which is not a reliable guide to ten times the volume | Model usage, hosting and review time tracked per item as volume grows |
| People | A handful of enthusiastic users | Everyone who touches the process, including those who liked the old way, with training and a channel for reporting problems |
Building AI into a defined workflow covers the wider design decisions behind these, from choosing the first process to measuring whether the result is working.
What does a short pilot to production checklist look like?
Before calling a pilot finished, confirm the following.
- A named business owner has reviewed the results against a baseline recorded before the pilot started.
- The test set includes real exceptions, not only clean examples.
- Integration with the systems the work actually runs in is built, not planned for later.
- Approval points are defined for every consequential action, with a named owner for the exception queue.
- Security and compliance reviewed the pilot before production access was requested, not after.
- Someone is named to monitor, support and improve the system once it is live. Use the AI automation handover checklist to verify their access, operating procedures and acceptance evidence.
- Cost per item at expected production volume has been estimated, not assumed from pilot volume.
How Kastling approaches moving from pilot to production
Kastling treats a pilot as part of the same engagement as production, not a separate project with its own rules. Development is tested against real business scenarios from the start, so the workflow, the data and the exceptions used in a pilot are the same ones the production system will face later. Before anything runs live, we agree with the business owner what success looks like, measured against a baseline, and we agree which actions the system can take on its own and which need a named person to approve.
Integration into existing systems is scoped from the beginning rather than added once a pilot proves the idea, and monitoring, support and running costs are discussed as part of the same proposal, under a separate agreement where they apply. Read more about AI Integration & Automation.
Human approval planner
Describe one AI task and see the level of human control it needs, with a control pattern your team can set up.
Questions
How long should an AI pilot run before we decide whether to scale it?
Long enough to see real volume and real exceptions, not a fixed number of weeks. Set the end date by the number of real cases the pilot has handled, including the awkward ones, and agree that number before it starts. A pilot that has not yet seen the range of cases the process actually produces is not finished, whatever the calendar says.
Can a pilot use production data, or does it have to wait for a formal rollout?
It should use production data from the start, with the same access controls and redaction rules that will apply once the system is live. A pilot built on sanitized sample data answers a different question than the one you actually need answered.
Who should own an AI pilot, IT or the business team?
The business team that owns the process, with IT and any AI specialists as delivery partners. A pilot owned by IT alone tends to prove the technology works and stop there, because nobody with authority over the workflow is accountable for what happens after the demo.
What is the most common reason a promising pilot never reaches production?
No one owns what happens after it works. The team that built the pilot moves to the next project, the workflow was never integrated into the systems people use daily, and the initiative quietly stops being a task anyone owns.
Should the pilot include the security review, or save it for later?
Include it from the start. A security or compliance review left until someone asks about production access can add significant delay to the project timeline, and the questions it raises, about data location, retention and access, are cheaper to answer before the pilot is built than after.