Key takeaways
- Pilots usually stall for operational reasons visible from day one: no named owner, no baseline, a demo on clean data and no plan for exceptions.
- A pilot designed to become production costs little more than one designed only to prove a point, if integration, a test set and an approval design are in scope from day one.
- Decide whether to scale, fix or stop against criteria agreed before the results come in, not after.
- Monitoring, access, support and cost per item all change between a pilot and production, and none of them are optional extras to add later.
- Naming who owns the system after launch, before the pilot starts, is one of the cheapest decisions that most affects whether it ever reaches production.
A pilot that impresses in a demo and a workflow that runs in production are different things, and the gap between them is where most AI initiatives quietly die. Some teams call this early stage a proof of concept, a limited test built to show an idea works before anyone commits to building it properly. The model worked. The steering group nodded. Then nothing changed about how the invoice, the ticket or the application actually moves through the business, because nobody built the parts a demo does not need: an owner, an integration, an approval design and a plan for what happens when the input is messy. This piece covers why that gap opens, how to design a pilot that can close it from day one, and how to decide, with evidence rather than optimism, whether to scale, fix or stop.
Why do AI pilots stall before they reach production?
Most pilots stall for operational reasons that were visible on day one, not because the model failed. Eight patterns account for nearly all of them.
- No named business owner. A pilot run by an IT or innovation team, with no one from the process itself accountable for the result, has no one to decide what happens next.
- No baseline. Without a measurement of how the work is done today, cycle time, error rate, hours spent, there is nothing to compare the pilot against, so the result becomes a matter of opinion.
- A demo on clean data. Ten tidy documents prove the model can succeed once, on data somebody picked. Live work sends the badly scanned page and the contract with two effective dates.
- No integration into the systems where the work happens. A pilot that runs in a separate tool, with someone copying results back into the ERP or CRM by hand, is not automation. It is a manual step with an AI assistant attached.
- No plan for exceptions and approvals. A pilot that only shows the happy path has not answered the question that decides whether it is safe to run unattended: what happens when the system is unsure.
- Security review left to the end. A pilot that reaches the security or compliance team only when someone asks about production access restarts the clock, often by months.
- No one to run it after launch. A pilot built by a vendor or a consultant, with no internal owner named for monitoring and support, has no path into daily operation, whatever the results.
- Costs unknown at scale. A pilot processing fifty items a day says little about what a thousand a day will cost in model usage and review time, and finance will ask before agreeing to scale.
According to McKinsey's 2025 survey of 1,993 participants across 105 countries, the share of organizations reporting AI use in at least one business function reached 88 percent, an increase of ten percentage points from the year before, yet only 7 percent of respondents said AI had been fully scaled across their organization (McKinsey). Adoption is no longer the hard part. Turning a working pilot into part of how the business runs is.
Gartner forecasts that over 40 percent of agentic AI projects will be canceled by the end of 2027 because of escalating costs, unclear business value or inadequate risk controls (Gartner). That is a forecast, not a count of completed failures, but it lines up with the reasons above: most agentic AI work today is still an early experiment, not a production system with a named owner and a plan.
How do you design a pilot that can become production from day one?
A pilot designed to prove a point and a pilot designed to become production are different projects, and the second one costs little more than the first. Seven decisions separate them.
- A real workflow, not a representative one. Pick the actual process a team runs today, with its actual volume and its actual exceptions, rather than a simplified version built to showcase the model.
- Real data from the start. Pull actual documents, tickets or requests from your own systems, including the awkward ones, rather than a curated sample.
- A success measure agreed before the pilot starts. State the target in the same units as your baseline, agreed with the business owner rather than the technical team.
- Integration in scope from day one. Even a lightweight connection to the ERP, CRM or ticketing system used every day tells you more than a standalone demo ever will, and building it later is a second project, not a formality.
- A test set built from real cases. The same fifty to a hundred documented examples used to test the system before launch double as the evidence for the scale decision. How to test AI before it touches your operations covers how to build one.
- The approval design, not only the workflow design. Decide which actions the system can take on its own, which need a named person to approve, and which stay with people entirely, before the pilot runs rather than after something goes wrong. The human approval planner works through this for one task at a time and returns a control pattern your team can set up.
- An owner for day two. Name who runs, monitors and improves the system after the pilot ends, before it starts. A pilot with no owner for what comes after is a demo with a longer runtime.
How do you decide whether to scale, fix or stop?
The end of a pilot is a decision point, not an automatic green light, and it works best against criteria agreed before anyone sees the results.
| Decision | When it applies | What has to be true |
|---|---|---|
| Scale | The pilot met its agreed target against baseline, exceptions route to a named owner, and cost per item holds up at expected volume | Integration, monitoring and support are scoped for daily operation, not only the pilot period |
| Fix | The pilot missed its target in a specific, explainable way, such as one document type or one exception category | The gap can be named, and closing it does not require redesigning the whole workflow |
| Stop | The pilot cannot reach an acceptable error rate on the cases that matter, the data needed does not exist in usable form, or no owner will take responsibility for it in production | Stopping now costs less than finding this out after a wider rollout |
Illustrative example: Consider a regional telecommunications provider piloting AI to draft replies to service activation requests. The pilot runs for six weeks against real tickets, with a named support lead reviewing every drafted reply before it is sent. At the end of the pilot, drafts need editing on a similar share of routine requests as the baseline the team recorded before starting, but replies to one specific request type, an address change bundled with a plan upgrade, are wrong often enough that the team routes that category to a fix list rather than calling the whole pilot a success or a failure.
Stopping is a legitimate outcome, not a failure to report quietly. A pilot that shows a workflow is not ready for AI, and explains why, has still done its job.
What changes between a pilot and production?
A pilot and its production version usually run on the same model, but almost everything around the model changes.
- Monitoring. A pilot gets watched by the people who built it. Production needs monitoring that runs without anyone remembering to look, with alerts when quality or cost drifts.
- Access. Pilot access is often broader than it should be, because speed mattered more than governance during the test. Production needs named service accounts, least privilege and a record of who can see what.
- Support. Someone has to own what happens when the system errors, when a source system changes its layout, or when the underlying model is retired or upgraded, and that person needs to be named and resourced.
- Cost at volume. Model usage, hosting and review time all scale with the number of items processed, and a cost per item measured in a pilot is not a reliable guide to cost at ten times the volume.
- Change management and training. A pilot has a handful of enthusiastic users. Production has everyone who touches the process, including the people who liked the old way, and they need training and a channel for reporting problems.
Building AI into a defined workflow covers the wider design decisions behind these, from choosing the first process to measuring whether the result is working.
What does a short pilot to production checklist look like?
Before calling a pilot finished, confirm the following.
- A named business owner has reviewed the results against a baseline recorded before the pilot started.
- The test set includes real exceptions, not only clean examples.
- Integration with the systems the work actually runs in is built, not planned for later.
- Approval points are defined for every consequential action, with a named owner for the exception queue.
- Security and compliance reviewed the pilot before production access was requested, not after.
- Someone is named to monitor, support and improve the system once it is live.
- Cost per item at expected production volume has been estimated, not assumed from pilot volume.
How Kastling approaches moving from pilot to production
Kastling treats a pilot as part of the same engagement as production, not a separate project with its own rules. Development is tested against real business scenarios from the start, so the workflow, the data and the exceptions used in a pilot are the same ones the production system will face later. Before anything runs live, we agree with the business owner what success looks like, measured against a baseline, and we agree which actions the system can take on its own and which need a named person to approve.
Integration into existing systems is scoped from the beginning rather than added once a pilot proves the idea, and monitoring, support and running costs are discussed as part of the same proposal, under a separate agreement where they apply. Read more about AI Integration & Automation.
Human approval planner
Describe one AI task and see the level of human control it needs, with a control pattern your team can set up.
Questions
How long should a pilot run before we decide whether to scale it?
Long enough to see real volume and real exceptions, not a fixed number of weeks. Most operational pilots need four to eight weeks once integration is in place, but a pilot that has not yet seen the range of cases the process actually produces is not finished, whatever the calendar says.
Can a pilot use production data, or does it have to wait for a formal rollout?
It should use production data from the start, with the same access controls and redaction rules that will apply once the system is live. A pilot built on sanitized sample data answers a different question than the one you actually need answered.
Who should own an AI pilot, IT or the business team?
The business team that owns the process, with IT and any AI specialists as delivery partners. A pilot owned by IT alone tends to prove the technology works and stop there, because nobody with authority over the workflow is accountable for what happens after the demo.
What is the most common reason a promising pilot never reaches production?
No one owns what happens after it works. The team that built the pilot moves to the next project, the workflow was never integrated into the systems people use daily, and the initiative quietly stops being a task anyone owns.
Should the pilot include the security review, or save it for later?
Include it from the start. A security or compliance review left until someone asks about production access usually adds months back onto the project timeline, and the questions it raises, about data location, retention and access, are cheaper to answer before the pilot is built than after.