Running AI in production on AWS and Azure

A working AI prototype is easy to like. It answers questions from a handful of test documents, runs on a developer's API key and costs very little. Then someone asks whether it can go live for 300 employees, or for customers, and the conversation changes: who can use it, which data it can see, what happens when it is wrong or unavailable, and who pays the bill. This guide covers what needs to be in place before an AI workload runs in production on AWS or Azure, written for the IT or operations leader who will own the outcome.

What changes when an AI prototype goes into production?

A prototype answers one question: can a model do this task well enough to be worth pursuing? Production answers a longer list. In this guide, production means a system that real people or processes depend on, using real business data, where a failure has a cost.

The model itself often changes least. Almost everything around it changes.

Illustrative example: Consider a regional distributor whose customer service team tested an assistant that answers order-status questions from ERP data. In the demo it performed well. Preparing it for production raised questions the demo never did: the ERP API slows down under bursts of requests, customers must only see their own orders, invoice data has to stay in the US, and agents need a way to take over when a shipment cannot be found. None of those questions were about the model.

Most of the work between demo and production is ordinary systems engineering: identity, integration, monitoring and ownership. It is also where most of the risk sits, which is why it deserves a plan before anyone sets a launch date.

What does a production AI architecture look like on AWS or Azure?

The shape is the same on both clouds; mostly the service names differ. It helps to think in layers, each with a clear job. AWS publishes a Well-Architected Generative AI Lens that covers these concerns (security, reliability, cost and operations) and describes its best practices in a cloud-agnostic way, so it works as a checklist whichever platform you choose.

Application layer

This is what people touch: a web app, a chat window, an API that another system calls, or a background job that reads incoming documents. It runs on ordinary cloud compute such as containers or serverless functions. Keep business rules here rather than inside prompts. Validation, permission checks and the list of actions the AI is allowed to take belong in code you can test.

Model access layer

Managed model services host the models so you do not run model servers yourself. On AWS that is Amazon Bedrock. On Azure it is Microsoft Foundry (formerly Azure AI Foundry), which includes Azure OpenAI. As of September 2026, both catalogs include models from OpenAI and Anthropic alongside other providers, so the choice between clouds rarely comes down to a single model. The factors that usually decide it are covered in AWS or Azure for AI workloads.

Call models through one internal interface or gateway rather than spreading a provider's SDK through the codebase. That single decision makes model upgrades, fallbacks and cost tracking much easier, as explained in model-agnostic AI architecture.

Retrieval and data layer

Most business AI needs your information, not just the model's general knowledge. Retrieval-augmented generation (RAG) is the common pattern: the application searches your documents or databases for relevant passages and passes them to the model along with the question. This layer includes a search index, often a vector index (a store that finds text by meaning rather than exact words), the embedding model that converts text for that index, and the jobs that keep it in step with source systems. Managed options include Amazon Bedrock Knowledge Bases and Azure AI Search. Whatever you use, carry permissions through: if an employee cannot open a file in SharePoint, the assistant should not quote it to them.

Integration layer

AI earns its place when it connects to the systems where work happens: ERP, CRM, ticketing, email and document management. Integrations can use direct APIs, message queues or managed connectors. Amazon AppFlow, for example, lists connectors for Salesforce, SAP OData, ServiceNow and Microsoft Dynamics 365, and Azure Logic Apps offers connectors for services such as Office 365, Dynamics and Salesforce. Reading data is the easier half. Any action that writes to a system of record should go through a narrow, documented operation, with human approval where the consequence warrants it. See connecting AI to your ERP, CRM and data warehouse for the detail.

Identity, access and secrets

People should sign in with company identity, and each workload should run under its own identity with only the permissions it needs. Microsoft's Foundry documentation describes keyless authentication with Microsoft Entra ID as a way to remove API keys entirely, and advises disabling key-based authentication once every consuming application supports it. On AWS, IAM roles do the same job, and IAM Identity Center can federate with Entra ID through SAML and SCIM when your workforce identity lives in Microsoft. Credentials that must exist, such as an ERP service account, belong in AWS Secrets Manager or Azure Key Vault rather than in code. Secrets Manager also supports automatic rotation schedules.

Logging, monitoring and evaluation hooks

You need two kinds of visibility. Operational monitoring covers errors, latency, throttling, token usage and cost, using tools such as Amazon CloudWatch and AWS CloudTrail on AWS, or Azure Monitor and Application Insights on Azure. Quality monitoring asks whether answers are still good. That is the job of evaluation hooks: a way to sample real inputs and outputs, score them, and add hard cases to your test set. Amazon Bedrock evaluations and Foundry evaluations can both score outputs against datasets you provide. Our guide to testing AI before it touches your operations explains how to build that test set.

Be deliberate about what you log. Amazon Bedrock model invocation logging is disabled by default, and once enabled it can capture full request and response data. Prompts often contain personal or confidential information, so those logs need the same access controls and retention rules as the source data.

Guardrails, cost controls and fallbacks

Guardrails filter harmful content and sensitive data in inputs and outputs: Amazon Bedrock Guardrails on AWS, and Foundry guardrails (previously called content filters) on Azure. Cost controls and fallbacks get their own sections below, because they are where production systems most often surprise their owners.

Should AI run in your cloud account or a partner's?

Before any build starts, decide whose cloud account the workload lives in. It sounds administrative, but it determines who controls access, who sees the bill, where logs sit and how hard it is to change partners later.

Running in your own AWS account or Azure subscription means you own the resources, the data and the billing relationship with AWS or Microsoft. A partner works through scoped roles that you grant and can remove. Model usage appears on your cloud invoice, where it may count toward commitments you already hold. On Azure, for example, eligible Marketplace purchases count toward a Microsoft Azure Consumption Commitment.

Running in a partner-hosted account can be quicker to start, particularly for a company with no cloud team or no existing account. The trade-off is dependence: you rely on the partner for access, data exports and continuity, so exit terms matter from the first day.

Two practices make either option safer. First, define the environment as infrastructure as code, meaning version-controlled templates that can recreate it in another account. Second, write the operating responsibilities into the agreement, because hosting a workload does not by itself include monitoring, backups, maintenance or incident response. If the systems your AI needs still run on premises or on aging servers, the account decision often belongs inside a wider legacy application migration plan.

How do you keep AI data secure and in the right region?

Start with the shared responsibility model. AWS describes its side as security of the cloud (the infrastructure) and the customer's side as security in the cloud: your data, identities, configuration and applications. A managed model service reduces what you run, not what you are accountable for.

What the platform does with prompts and outputs

Check this for each model you plan to use, because terms can differ within one platform.

• Amazon Bedrock runs models in AWS-operated deployment accounts that model providers cannot access, so providers do not see your prompts or completions.
• Bedrock also lets you set a data retention mode, and some newer models require AWS to retain inputs and outputs for up to 30 days for human review by AWS as a condition of access.
• For Foundry Models sold by Azure, including Azure OpenAI, Microsoft states that prompts and completions are not available to OpenAI or other model providers and are not used to train foundation models without your permission.
• Claude models in Foundry come in two versions. Some run on Azure infrastructure end to end; others run on Anthropic's infrastructure outside Azure. Check which version you deploy.

Where prompts are processed

Picking a region is not the whole answer, because deployment options can route inference elsewhere.

• On Azure, Global deployment types may process prompts in any Azure region, Data Zone types keep processing within the US, EU or Asia Pacific zone, and Standard or Regional Provisioned types keep it within your chosen Azure geography. Data stored at rest remains in your designated geography for every type.
• On AWS, geographic cross-Region inference keeps processing within a geography such as the US or EU, while global cross-Region inference can route requests to any supported commercial Region.

Network paths and least privilege

Keep traffic private where you can, using AWS PrivateLink interface endpoints for Bedrock or private endpoints for Foundry resources. Then give the AI no more access than the person it works for. If an agent can call a tool, the tool should check permissions itself rather than trusting the model to stay within bounds. For policies, data classification and approval flows, see data security and governance for AI in the enterprise.

How do you control model usage costs?

Model usage is billed mainly by tokens, the small chunks of text a model reads and writes. Cost grows with the model you choose, the length of prompts (retrieved documents add a lot), the length of answers, request volume, and hidden multipliers such as retries and agents that call a model several times for one task.

Budget alerts alone will not contain this. Microsoft's Cost Management documentation states that when a budget threshold is reached, resources are not affected and consumption is not stopped. AWS notes that Budgets notifications can arrive after costs have already exceeded the threshold. Hard limits have to live in your application or gateway.

A practical cost checklist:

1. Use the smallest model that passes your evaluation set, and retest when prices or models change.
2. Cap output length and the number of model calls an agent can make per task.
3. Trim retrieved context to what the answer actually needs.
4. Move work that is not urgent to batch processing. As of September 2026, Amazon Bedrock prices batch inference 50% below on-demand for select models, and Azure's Global Batch runs at 50% less than Global Standard.
5. Choose deployment options with residency in mind. Bedrock lists global cross-Region inference at approximately 10% savings compared with geographic inference, but it only suits data without location restrictions.
6. Attribute spend to teams and applications, for example with Bedrock application inference profiles, request metadata set in a shared client, or per-consumer token metrics from Azure API Management.
7. Enforce per-application token limits at a gateway. Azure API Management's token limit policy and the per-key budgets in the open-source LiteLLM proxy are two ways to do it.
8. Reserve capacity only for steady, high-volume or latency-sensitive traffic, and only after measuring real usage.

What reliability patterns does production AI need?

Model endpoints behave differently from a typical database. Responses can take many seconds, capacity is shared with other customers, and quotas are measured in tokens per minute as well as requests per minute. Plan for these patterns before launch.

Timeouts. Set a timeout on every model call and an overall time budget for each user request. A person waiting 90 seconds for an answer has usually given up already.

Retries with backoff. When a platform throttles a request, retry after a growing delay rather than immediately. Microsoft's guidance for Claude models in Foundry, for example, is to handle 429 (too many requests) responses with exponential backoff. Cap the number of retries, since each attempt costs time and possibly tokens.

Rate limits and quotas. Know your quotas before launch and request increases early. Microsoft notes that 429 responses can appear even when usage metrics look below quota, and that latency can rise when traffic exceeds your usage tier. Protect shared quota with per-application limits so a batch job cannot starve a customer-facing assistant.

Fallbacks. Decide in advance what happens when the primary model or region is unavailable. Microsoft states that Foundry itself does not provide automatic failover or disaster recovery, and recommends a gateway such as Azure API Management, with load balancing and a circuit breaker (a rule that stops sending traffic to a failing endpoint until it recovers), in front of deployments in more than one region. A fallback model is only safe if it has passed the same evaluation set as the primary.

Human handoff. The most important fallback is a person. Define when the AI hands over: low confidence, missing data, a failed integration or a customer asking for a human. The handoff should land in a named queue with the context attached, and the team should be able to run the process manually while the AI path is switched off.

How should you deploy and change AI systems safely?

Treat everything that changes the system's behavior as a release: code, prompts, model versions, retrieval settings, tool definitions and guardrail configuration. A one-line prompt edit can change outputs as much as a code change.

A sound release process includes:

• Separate development, test and production environments, ideally in separate accounts or subscriptions.
• Infrastructure as code and an automated pipeline, so every environment is built the same way.
• An evaluation run before each release, where a drop in quality blocks the release.
• A staged rollout to a small group first, with the previous version ready to restore.
• A short note to users on what changed and what to do differently.

Model retirement is a change you can schedule, because both platforms publish their policies. On Microsoft Foundry, generally available models retire 18 months after launch (12 months for models from Anthropic, DeepSeek, Fireworks and Mistral AI), subscription owners receive notice at least 60 days ahead, and retirement dates cannot be extended. Standard deployment types can also be upgraded automatically at retirement unless you configure otherwise, which means a model change could reach production outside your release process. On Amazon Bedrock, models launched from September 7, 2026 carry a Legacy period of 6 months or 45 days before end of life, after which requests fail and migration does not happen automatically. Record these dates with an owner, and run the replacement through your evaluation set well before the deadline.

Which operating responsibilities need a named owner?

Hosting and operating are different jobs. A workload can run in a well-built account for months with nobody watching it, until the day it matters. Agree these responsibilities in writing before launch, whether they sit with your team, a partner under a maintenance agreement, or a mix of both.

One item deserves its own mention: a documented off switch. Every production AI workflow should have a tested way to disable the AI path and return to manual handling without deploying new code.

How does Kastling approach AI in production on AWS and Azure?

Kastling's AI & Cloud Infrastructure service covers cloud architecture and deployment on AWS or Azure, AI application deployment, data connections and pipelines, and migration of agreed workloads. The approach follows the principles in this guide:

• Understand the work and the existing systems before choosing technology, starting with a discovery call and, where the scope warrants it, a separately scoped paid audit.
• Connect existing systems rather than replacing them.
• Deploy in your AWS account or Azure subscription, or in Kastling's accounts, with account ownership, access, operating costs and responsibilities set out in the proposal. Usage in Kastling-hosted accounts is billed on a pay-as-you-go basis.
• Test against real business scenarios before launch, and keep a named person in control of consequential actions.
• Agree data access and ownership up front.
• Scope monitoring, backups and incident response explicitly rather than assuming hosting includes them. Ongoing maintenance is available under a separate agreement.

The team is founder-led, with AWS and Azure specialists brought in as the work requires.