Monitoring AI and LLM applications in production: what to track

For a company of roughly 20 to 200 people running its first AI workflows, launch day is when the work looks finished, and it is the day monitoring begins. A conventional application breaks loudly: a server goes down, a page errors, someone is paged. An AI workflow can keep running at full speed while its answers slowly get worse, its costs creep up, or the approval queue behind it fills with items nobody reads. The plan for noticing those things should exist before the first real user does.

Why is monitoring AI different from monitoring other software?

Traditional monitoring asks whether the system is up and fast. AI monitoring has to ask a second question: whether the output is still correct. The same code, the same prompt and the same model can produce different quality next month because the documents changed, a new type of request arrived, or the provider quietly updated a model.

Microsoft's documentation on AI observability puts the scope plainly. It describes observability as the ability to monitor, understand and troubleshoot AI systems, using signals such as evaluation metrics, logs, traces and model outputs to gain visibility into performance, quality, safety and operational health. The useful word there is quality. Availability is necessary and not sufficient.

If a vendor or team proposes an AI workflow with no answer to "how will we know when it gets worse," it is not ready for production.

What should you log for every LLM or AI request?

Log enough to reconstruct any decision after the fact. When a customer disputes an answer or an auditor asks why an invoice was routed a certain way, the log is the only evidence.

For each request, capture:

• Who and what triggered it: the user or system identity, the workflow, the timestamp and a request ID that follows the task through every step.
• Which model and version ran: the provider, model identifier and prompt version, so a change in behavior can be tied to a change in configuration.
• Inputs and outputs: the prompt, any retrieved documents or their identifiers, and the response.
• Tool calls and actions: every call the AI made to another system (a lookup in your ERP, a draft in your CRM, a message sent) with the parameters and the result. For anything that can change data, the log should show who approved it.
• Token counts, latency and errors: the raw material for cost and performance metrics.
• The human outcome: whether a reviewer accepted, edited or rejected the result.

Both major clouds provide a starting point. Amazon Bedrock's model invocation logging can collect the full request data, response data and metadata for supported calls in an account and Region, delivered to CloudWatch Logs or Amazon S3. It is disabled by default, so someone has to switch it on deliberately. Each record includes the model ID, the identity that made the call and the input and output token counts, and callers can attach their own request metadata, such as a team or environment tag, to break down usage. Microsoft Foundry integrates with Azure Monitor Application Insights for operational metrics, token consumption, latency, error rates and quality scores.

If your application spans several systems, tracing helps. OpenTelemetry, the open standard for collecting traces and metrics, has a set of semantic conventions for generative AI covering spans, metrics and events for AI clients, and Microsoft describes its own tracing as built on OpenTelemetry standards. Using the standard keeps your records portable across tools and providers.

How do you handle personal data in AI logs?

Logs of prompts and responses are some of the most sensitive data you will ever store, because they contain whatever people typed or pasted in. Plan for that before you enable them.

1. Decide what is allowed in the logs. Agree with your privacy or security lead whether raw text may be stored, for how long, and where.
2. Mask on the way in. AWS describes CloudWatch Logs data protection policies that audit and mask sensitive data as it is ingested, with managed identifiers for credentials, financial information, personal and health information. A caveat from the same documentation: masking applies to events ingested after the policy is set, not to earlier ones. Turn it on first, then start logging.
3. Limit who can unmask. Only a small, named group should be able to view the original text, and that access should itself be logged.
4. Set retention. Keep raw logs only as long as investigations and your obligations require, and keep aggregated metrics longer.
5. Keep logs in your own account. The data and its retention rules should sit under your control, not only in a vendor's dashboard.

How do you measure whether the answers are still good?

Quality does not appear on an infrastructure dashboard, so you have to create the measurement. Four methods work together.

• Sampled human review. Pull a regular sample of real outputs and have the workflow owner mark them correct, partly correct or wrong. Over-sample low-confidence results and high-value cases. This is the only measure that catches errors nobody thought to test for.
• Scheduled evaluation reruns. Keep the frozen test set from before launch and run it on a schedule, and again after every change to the prompt, model or data source. Microsoft's guidance lists scheduled evaluation using test datasets to detect system drift as part of post-production monitoring. Our guide to testing AI before it touches your operations explains how to build that set.
• Continuous evaluation on live traffic. Automated scoring of a sampled share of production requests, using checks such as whether an answer is grounded in the retrieved documents. Foundry offers this built in, and similar checks can be written in ordinary code.
• Behavioral signals. Where a human works with the AI, their behavior is data. Track the override rate (how often reviewers change the AI's result), the rejection rate, the share of items the AI flags as low-confidence, and complaints or corrections from downstream teams.

A rising override rate is often the earliest warning in practice, because the people using the system notice problems before an automated score does.

What is drift, and how do you spot it?

Drift is a gradual change in either the inputs or the outcomes that makes the system less accurate than it was at launch. In practice it comes from four places:

• The inputs change. New suppliers, new document layouts, a new product line, a new language, a change in how customers phrase requests.
• The knowledge changes. Policies are revised and the retrieval index falls behind, or documents are added that contradict older ones.
• The model changes. Providers retire and replace models on published schedules, and replacements do not behave identically.
• The business changes. Rules, thresholds and approval limits move, but the instructions given to the AI do not.

You spot drift by comparing against a baseline. Track the distribution of request types, the share of low-confidence results, the override rate and the scheduled evaluation scores over time, and flag movement beyond an agreed range. Record every model retirement date in the same calendar as your review dates, so a retirement notice never arrives as a surprise.

Illustrative example: Consider an operations team that uses AI to read supplier invoices and prepare entries for approval. The workflow passes testing and runs well for two months. Then a major supplier switches to a new invoice layout. Nothing errors and latency is unchanged, but the share of invoices flagged low-confidence climbs and reviewers start correcting the same field repeatedly. An alert on override rate fires, the owner pulls a sample, finds the new layout, adds examples to the test set and updates the instructions. Without that alert, the corrections would have continued as unrecorded extra work until someone asked why the team was busier.

What metrics should you track and alert on for AI in production?

Eight signals are worth a threshold on an AI workflow. Set a threshold for each, agree who is told, and decide in advance what counts as an emergency.

How should you track AI cost per task?

Track token cost per completed task, not total spend, because total spend rises legitimately with volume, while a rising cost per task points to longer prompts, more retries or a workflow stuck in a loop. Group usage by workflow or team using request metadata or the calling identity, so a surprise has an owner.

Why does approval queue age matter for AI monitoring?

Approval queue age shows whether the human step is keeping up. If a person must approve an action, the queue is part of the system. An AI that drafts faster than people can review does not save time; it builds a backlog and invites rubber-stamping. Alert on how old the oldest waiting item is, and treat a growing queue as a staffing or design problem.

What should the incident response and rollback plan include?

Decide what you will do before you need to do it. A one-page runbook is enough, as long as it exists and someone has rehearsed it.

1. Define severity. Wrong answers that reached customers, personal data exposed, actions taken that should not have been, versus a slow response or a single failed task.
2. Know the stop switch. How to pause the automation, route work back to the manual process, and who is allowed to do so without asking permission. Keep the manual process alive and documented long enough to make this real.
3. Be able to roll back. Keep prior versions of prompts, configuration and model selection so the last known good state can be restored in minutes, not rebuilt from memory.
4. Contain and notify. Identify which records the AI touched using the logs, correct them, and tell the affected teams and, where rules require it, customers or regulators.
5. Learn. Add the failing case to the test set, fix the cause, and note what the monitoring missed.

For the handover side of this, including who takes over when the build team steps back, see the AI automation handover checklist and runbook template.

Who owns AI monitoring?

Unowned alerts are the same as no alerts. Name people, not teams, for each of these:

• Business owner: quality thresholds, the review sample, and the decision to pause or keep running.
• Technical owner: availability, latency, errors, integrations, model retirement dates and cost alerts.
• Reviewer or reviewers: the people who do the sampled reviews, with time protected for it.
• Security or privacy lead: log retention, raw text access and incident notification.

Review the numbers on a set schedule, weekly at first, then monthly once the workflow is stable. A monitoring dashboard no one looks at is decoration.

A starting checklist

1. Invocation and tool-call logging is on, with request IDs, model and prompt versions.
2. Sensitive data is masked at ingestion, raw text access is restricted and retention is set.
3. A frozen test set reruns on a schedule and after every change.
4. A sample of live outputs is reviewed by a person, with the sample size agreed.
5. Override rate, low-confidence share and error rate have baselines and thresholds.
6. Token cost per completed task is tracked by workflow.
7. Approval queue age has a limit and an owner.
8. A runbook covers pausing, manual fallback, rollback and notification.
9. Every alert and metric has a named owner and a review date.
10. Model retirement dates are tracked alongside the review calendar.

How Kastling approaches monitoring

Monitoring is part of the operating design in AI & Cloud Infrastructure work: cloud architecture and deployment on AWS or Azure, AI application deployment, and the data connections behind it. The wider picture of running AI on these platforms is in our guide to running AI in production on AWS and Azure.

Hosting does not automatically include monitoring, backups or incident response, so Kastling sets out in the proposal who is responsible for each, whether the workload runs in the client's own AWS account or Azure subscription or in Kastling-hosted accounts. Solutions are tested against real business scenarios before go-live, data access and ownership are agreed up front, and ongoing maintenance is available under a separate agreement that defines what is included. The team is founder-led, with AWS and Azure specialists brought in as the work requires.